*This short but very useful tip was submitted by Charly Wargnier.*
When doing SEO changes for large scale companies, implementing a proper robot.txt is crucial. I will not go back to the robot.txt definition bla, bla… millions have done that before me.
No, instead, just a simple formula to use whenever the geek inside you has a Robot.txt Writer’s block! So, type “inurl:robots.txt filetype:txt” and, ta-daa! See what the big names are doing.
You will find the robots.txt file from Google, Wikipedia, WebmasterWorld, the White House, Microsoft, W3.org, Facebook, IBM, Amazon, Ebay, New York Times, CNN, YouTube, etc.
Have Geek Fun.
Latest posts by DST Contributor (see all)
- Survival Guide to Multilingual SEO - May 14, 2013
- Five Killer Link Bait Tips That Can Provide You With ‘Passive Marketing’ - May 11, 2013
- 5 US SEO Events to Visit to Spend a One Month Vacation with Purpose - May 9, 2013