How to Avoid the Robots.txt Writer’s Block!
*This short but very useful tip was submitted by Charly Wargnier.*
When doing SEO changes for large scale companies, implementing a proper robot.txt is crucial. I will not go back to the robot.txt definition bla, bla… millions have done that before me.
No, instead, just a simple formula to use whenever the geek inside you has a Robot.txt Writer’s block! So, type “inurl:robots.txt filetype:txt” and, ta-daa! See what the big names are doing.
You will find the robots.txt file from Google, Wikipedia, WebmasterWorld, the White House, Microsoft, W3.org, Facebook, IBM, Amazon, Ebay, New York Times, CNN, YouTube, etc.
Have Geek Fun.
The guest post is by Charly Wargnier, SEO Head at the London digital agency Euston Digital. You can follow their SEO tips and tricks on ED’s Blog, or their Twitter here
9 Responses to “How to Avoid the Robots.txt Writer’s Block!”
Recent Comments
- Nijin @blogseoads.com on Search Engine Optimization Gone Bad
- winona on Social Media Marketing for Real Estate (Infographic)
- Dipak Rajyaguru on Link Evaluation Survey 2012
- Nick Stamoulis on Search Engine Optimization Gone Bad
- XNUMERIK on Importance of NoFollow Links In Driving Traffic
Friends and Partners
Tags
Archives
- April 2012
- March 2012
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
I always used robot.txt to avoid some of the crawling pages of my website. This one is really great. Thanks for info
Hi Charly,
We’re currently implementing a new site for aalabels and this piece of advice will be a great one for our Devteam! Bookmarked!
Nice one! I wouldn’t need to go into that for my site but this is quite fun too!
And The White House doesn’t seem very concerned about robot.txt!
I have not used the robots.txt much and have had great SEO results, great reminder to be doing this as standard practice.
Is there an easy way to set up a robot.txt?
You just create the robots.text file, David, and then upload to the root folder of your domain. That is job done.
This article is totally misleading. You should avoid blocking bots accessing pages via robots.txt: http://www.youtube.com/user/GoogleWebmasterHelp#p/a/u/2/CJMFYpYQZ0c
I found the time to go into more details:
Don’t block stuff in robots.txt when you want to steer search engine indexing, otherwise you can create PageRank sinks.
Never ever try to steer indexers in robots.txt until you really, really can’t avoid it. Not even duplicates. Especially not duplicates.
For Google, Bing and Yahoo, better use X-Robots or Meta Robots Tags directives “noindex,nosnippet,noarchive”.
Thanks for the tip. Sometimes its easier to learn by example.