Skip to content

SEO Crawler

A free, fast & flexible SEO website crawler to help identify technical SEO issues. Crawl up to 350 URLs for free!

If you only want to crawl a sub-set of URLs
If you want to exclude a sub-set of URLs. For example \?.*page=.*$
If you want to remove parameters from URLs. For example \?.*sessionid=.*$
Check for the presence of text within the HTML; for example a Google Analytics profile ID

Crawler features:

  • It's fast*. View real-time feedback of your crawl.
  • Crawl on the move - runs on mobile phones & tablets including iPad, iPhone and more.
  • Crawl up to a maximum of 350 URLs for free
    • Use advanced filters to customize your crawl to sub-sections of your website
  • Web reports show:
    • URL
    • Page title
    • HTTP status code
    • Details of redirect chains
    • Meta robots details
    • Followed external links & total links on-page
    • rel=canonical URLs
    • Any automatically detected SEO issues
  • An Excel-compatible TSV report can be exported after the crawl showing:
    • All of the above, plus:
    • Meta description
    • Total external links
    • Meta keywords
    • Meta refresh
    • rel=author details
    • rel=alternate (hreflang & media) details
    • rel=next/prev details
  • Use regular expressions to customize URL patterns to include, ignore, or strip parameters/folders out of the crawl.
  • Search for content within a site's pages using regular expressions - for example, use UA-123456-1 to look for presence of a Google Analytics Profile ID
  • Respects robots.txt rules & rel=nofollow
  • Test how your site behaves with different user-agents such as Desktop, Mobile, Tablet and search bots
  • Identify duplicate page titles, duplicated meta tags, server errors, redirect chains and more.

* Speed is limited to avoid impact on server load

© rob hammond | say hi | SEO Consultancy