*This short but very useful tip was submitted by Charly Wargnier.*
When doing SEO changes for large scale companies, implementing a proper robot.txt is crucial. I will not go back to the robot.txt definition bla, bla… millions have done that before me.
No, instead, just a simple formula to use whenever the geek inside you has a Robot.txt Writer’s block! So, type “inurl:robots.txt filetype:txt” and, ta-daa! See what the big names are doing.
You will find the robots.txt file from Google, Wikipedia, WebmasterWorld, the White House, Microsoft, W3.org, Facebook, IBM, Amazon, Ebay, New York Times, CNN, YouTube, etc.
Have Geek Fun.