Invisible side of a Website  | |  Some websites have a file called "robots.txt" in it's root. The file allows or disallows the search engine report the results to you that the website owner don't want to tell you about. But do you know that still you can view these results by typing/robots.txt after the website address; for example, If you want to know what Microsoft hides from you type http://www.microsoft.com/robots.txt and you will find the list of directories that Bill Gates want to hide from you.
| |
| |
| | | | Washington Scholarships Tap Into $7 Billion In Scholarship Funds. Get Free Step-By-Step Guide. FreeEducationGuide.com
| Hotel in Fairfax, VA Official Site- Hyatt Fair Lakes. Across from Fair Oaks Mall. Convenient access to Dulles Airport and Washington DC. Book Online for Hyatt.com's Rate Guarantee. fairlakes.hyatt.com
| Bartending School Maryland, DC, Virginia Full Service Bar School always open/with abundant job leads. www.bartending-school.com
|
|
| | | | | | 1.  cmdr001 (291) | 3 months ago |  It's funny actually because I had no idea of that and I didn't quite bothered to search up the information, however, I do have an Apache server up and running and I found weird that something had been looking for a robots.txt file.
Hardly could be a coincidence that someone was seeking such a file on my machine since there are only specific file links to it, but I didn't quite bothered with trying to see what it could be. They got nothing and the access is extremely restricted.
Still, it's a heads up and tells me that some web crawler bots do respect privacy in some way, but I guess most will just ignore that file altogether.
| |
| |
|
|
| kurtaxl (97) | 3 months ago | hehehe, at this mine too, I'm still amaze what this robots.txt, there's nothing impossible in cyber world! | |
|
| | Washington Scholarships Tap Into $7 Billion In Scholarship Funds. Get Free Step-By-Step Guide. FreeEducationGuide.com | add comment |
|
| | | | | |
| Washington D.C.'S Premier Entertainers
D.C.'s Top Comedy Magician Team. Exciting Dove And Stage Illusions.
www.speedthro.com | |
Used Honda Civic Sedan at Yahoo!
Certified Used Cars in Washington, DC. Shop for a Honda Used Car.
www.honda.com/yahoo | |
|
|
|
|
|
