SEO Crawler

A free, fast & flexible SEO website crawler to help identify technical SEO issues. Crawl up to 350 URLs for free!

Seed URL:

Include:

If you only want to crawl a sub-set of URLs

Ignore:

If you want to exclude a sub-set of URLs. For example \?.*page=.*$

Remove:

If you want to remove parameters from URLs. For example \?.*sessionid=.*$

Search for:

Check for the presence of text within the HTML; for example a Google Analytics profile ID

User Agent:

Crawl sub-domains:

It's fast*. View real-time feedback of your crawl.
Crawl on the move - runs on mobile phones & tablets including iPad, iPhone and more.
Crawl up to a maximum of 350 URLs for free
- Use advanced filters to customize your crawl to sub-sections of your website
Web reports show:
- URL
- Page title
- HTTP status code
- Details of redirect chains
- Meta robots details
- Followed external links & total links on-page
- rel=canonical URLs
- Any automatically detected SEO issues
An Excel-compatible TSV report can be exported after the crawl showing:
- All of the above, plus:
- Meta description
- Total external links
- Meta keywords
- Meta refresh
- rel=author details
- rel=alternate (hreflang & media) details
- rel=next/prev details
Use regular expressions to customize URL patterns to include, ignore, or strip parameters/folders out of the crawl.
Search for content within a site's pages using regular expressions - for example, use UA-123456-1 to look for presence of a Google Analytics Profile ID
Respects robots.txt rules & rel=nofollow
Test how your site behaves with different user-agents such as Desktop, Mobile, Tablet and search bots
Identify duplicate page titles, duplicated meta tags, server errors, redirect chains and more.

* Speed is limited to avoid impact on server load