Crawl Your Site for Broken Links, Errors and Duplicate Content

One very overlooked part of the entire SEO mix is making sure that your site does not have broken outbound (or internal) links which either link to error pages, or do not work at all. Furthermore, if your site delivers error pages or links to non-existent pages or files on your server, then search engines like Google are going to consider your site as being “under construction“, therefore not being useful or relevant to the human user.

Your site can have all of the optimized content, titles and headers in the world, but if it is not functioning correctly, then it will not rank to the best of its ability. Working on client SEO accounts, I have run into numerous issues where fixing some internal broken links, outbound links and making sure all of the files on the server are working have boosted a site ranking from pages 3 or 4, to page one.

My favorite tool for checking this information is Xenu’s Link Sleuth.

Xenu is usually the first step I take in on-site SEO research and identifying issues. I like to get these issues tackled from the beginning and when working with an IT team on a client project, handing them a 10 page error report usually puts them in their place, and show’s them that you’re serious about your technical SEO.

Xenu’s Link Sleuth checks Web sites for broken links. Link verification is done on “normal” links, images, frames, plug-ins, backgrounds, local image maps, style sheets, scripts and java applets. It displays a continously updated list of URLs which you can sort by different criteria.

Xenu spiders a website in a similar fashion that a search engine will and delivers a report which looks at :

• Broken links on the site which send the spider to error messages
• Duplicate content issues such as similar title tags and URL structure
• Broken files such as images and multimedia content whcih are not loading correctly
• Images which do not include alt attributes (which can be helpful to SEO)
• Identifing files and images which may effect page load time
• Links to server redirected pages or 301 redirects which you can change on your site to link to the real page instead of the redirect command

In addition to checking for links via Xenu’s Link Sleuth, I also recommend doing a basic duplicate content and header error diagnosis. Sure, if you are using Google Webmaster Tools, this can be done easily, but we always don’t have access to Google Webmaster Tools, especially when working on 3rd party sites or performing competitive research.

One free tool which can be used to check for common duplcate content issues is the Virante Duplicate Content tool.

Virante’s tool diagnoses the following :

• Common www vs non-www duplicate content issue by checking the headers returned by both versions of the url, the current cache in google, and possible PR dispersion.
• Common default page error where both the / and /index.html (or other default page) return 200/OK headers.
• Incorrect 404 pages which deliver a 200/OK Header and,
• Supplemental pages in the Google index

By using tools like these to identify errors on your website, you can enhance your SEO and rankings substantially, especially if you have any critical errors which are keeping your site from ranking properly in the search engines.

What are some of your favorite tools for checking broken links, files and duplicate content issues? Please feel free to share them in the comments (and get a do-follow link back to your site!)