Crawl Your Site for Broken Links, Errors and Duplicate Content
One very overlooked part of the entire SEO mix is making sure that your site does not have broken outbound (or internal) links which either link to error pages, or do not work at all. Furthermore, if your site delivers error pages or links to non-existent pages or files on your server, then search engines like Google are going to consider your site as being “under construction“, therefore not being useful or relevant to the human user.
Your site can have all of the optimized content, titles and headers in the world, but if it is not functioning correctly, then it will not rank to the best of its ability. Working on client SEO accounts, I have run into numerous issues where fixing some internal broken links, outbound links and making sure all of the files on the server are working have boosted a site ranking from pages 3 or 4, to page one.
My favorite tool for checking this information is Xenu’s Link Sleuth.
Xenu is usually the first step I take in on-site SEO research and identifying issues. I like to get these issues tackled from the beginning and when working with an IT team on a client project, handing them a 10 page error report usually puts them in their place, and show’s them that you’re serious about your technical SEO. 
Xenu’s Link Sleuth checks Web sites for broken links. Link verification is done on “normal” links, images, frames, plug-ins, backgrounds, local image maps, style sheets, scripts and java applets. It displays a continously updated list of URLs which you can sort by different criteria.
Xenu spiders a website in a similar fashion that a search engine will and delivers a report which looks at :
- Broken links on the site which send the spider to error messages
- Duplicate content issues such as similar title tags and URL structure
- Broken files such as images and multimedia content whcih are not loading correctly
- Images which do not include alt attributes (which can be helpful to SEO)
- Identifing files and images which may effect page load time
- Links to server redirected pages or 301 redirects which you can change on your site to link to the real page instead of the redirect command
In addition to checking for links via Xenu’s Link Sleuth, I also recommend doing a basic duplicate content and header error diagnosis. Sure, if you are using Google Webmaster Tools, this can be done easily, but we always don’t have access to Google Webmaster Tools, especially when working on 3rd party sites or performing competitive research.
One free tool which can be used to check for common duplcate content issues is the Virante Duplicate Content tool.
Virante’s tool diagnoses the following :
- Common www vs non-www duplicate content issue by checking the headers returned by both versions of the url, the current cache in google, and possible PR dispersion.
- Common default page error where both the / and /index.html (or other default page) return 200/OK headers.
- Incorrect 404 pages which deliver a 200/OK Header and,
- Supplemental pages in the Google index
By using tools like these to identify errors on your website, you can enhance your SEO and rankings substantially, especially if you have any critical errors which are keeping your site from ranking properly in the search engines.
What are some of your favorite tools for checking broken links, files and duplicate content issues? Please feel free to share them in the comments (and get a do-follow link back to your site!)
88 Responses to “Crawl Your Site for Broken Links, Errors and Duplicate Content”
Recent Comments
- Lioniel Patricia on FTC Investigating Google for Possible Anti-Trust and Privacy Issues Over Google+
- Nick Stamoulis on Google Personalized Search: We Know Where You Live…
- Gerald Weber on Real People, Real SEO – Welcome to MySEOCommunity
- Julie Joyce on Real People, Real SEO – Welcome to MySEOCommunity
- Kreditkunde on 5 Reasons Why MyBlogGuest is My Best Find after 5 Years as Webmaster
Tags
Archives
- February 2012
- January 2012
- December 2011
- November 2011
- October 2011
- September 2011
- August 2011
- July 2011
- June 2011
- May 2011
- April 2011
- March 2011
- February 2011
- January 2011
- December 2010
- November 2010
- October 2010
- September 2010
- August 2010
- July 2010
- June 2010
- May 2010
- April 2010
- March 2010
- February 2010
- January 2010
- December 2009
- November 2009
- October 2009
- September 2009
- August 2009
- July 2009
- June 2009
- May 2009
- April 2009
- March 2009
- February 2009
nice tool, thanks so much for sharing goog job.
thanks.. Xenu is a great tool
Hey its really helpful for webmasters. Nice info. Thanks
it’s really good information but i want to know about the effects.
Thanks man
it’s really good information but i want to know about the effects.
Hallo everybody.
Very creative ways to use Twitter.
I´ll do it also.
Greetings
it’s really good information but i want to know about the effects.
Xenu is the best! I love the ability to export the data into Excel where I can manipulate it any way that I want.
thanks for nice informations. i liked your site too much
it’s too useful
Great Thanks !!
YEs very good text.
Forever site
Hmm…Anlamad?m ama deneyelim..
This a great tool just what I need. I noticed the error 404 on my stats fro crawls but never knew what to do about it. This is great, now I can check for broken links. Thanks for this great post.
yeha nice article dude
it’s really good information but i want to know about the effects.
Multicast Wireless is a mission-based, cutting edge, progressive multimedia organization located in Huntsville, Alabama
This is great article. Thank you very much for this post.
Thanks for letting us know about Xenu. It’s already been a big help!
Being a newborn in the world of SEO, and I find the info in this article easy to digest and beneficial. Thanks Hugo!
Being a newborn in the world of SEO, and I find the info in this article easy to digest and beneficial. Thanks!
This is a great tool. Sometimes I worry about my competitors finding this excellent site!
All Indian T.v Serials Available And Watch Live World Cup 2011 Only On
online desi videos,indian tv serials,dramas Watch Daily Episode
it’s too useful
Sara Technologies Pvt. Ltd is a global provider of high quality and cost-effective Information Technology Services. We offers a range of expertise aimed at helping customers re-engineer and re-invent their businesses to compete successfully in an ever-changing marketplace.
Sara Technologies Pvt. Ltd is a global provider of high quality and cost-effective Information Technology Services.
At være en nyfødt i verden af ??SEO, og jeg finder de info i denne artikel er let at fordøje og gavnlige, tak.
I agree “Sara Technologies Pvt. Ltd is a global provider of high quality and cost-effective Information Technology Services. We offers a range of expertise aimed at helping customers re-engineer and re-invent their businesses to compete successfully in an ever-changing marketplace.”