Website crawler
Crawl your website and discover its pages
Almost nobody knows how many pages their website really has. Orphans linger after a migration, a paginated series runs longer than anyone remembers, a whole section sits behind a link nobody updated. A crawler answers that by walking the site the way a search engine does — start at one address, follow what it finds, and keep a ledger of everything it saw.
Where the crawler finds your pages
Five sources feed the queue, and each discovered URL records which one produced it: the address you submitted, the XML sitemaps (including sitemap index files, followed to their children), the robots.txt declarations, the site navigation and footer, and ordinary links inside the pages already read. A page reachable through several of those is a page nobody has to worry about; one that appears in a sitemap and nowhere else usually is.
The order is decided, not accidental
When a site holds more URLs than the plan allows, which pages get read matters more than how many. Each URL is given a priority from observable facts alone — the depth it sits at, where the link was found, its anchor text, how many distinct internal pages point at it, and whether its path looks like a paginated series. The result is repeatable: the same site crawled twice picks the same pages, and the reason each one was chosen can be stated.
Every page that was not read is named, with the reason
Discovered, queued, fetched, rendered, analysed, blocked, challenged and failed are counted separately, and the URLs still waiting when the crawl stopped are counted too. A page refused by robots.txt, held behind a login, answering with a bot challenge, timing out or redirecting off-site is recorded with that exact reason rather than quietly dropped. A crawl report that only shows what worked is telling you about a different website than yours.
URLs are compared, not just collected
Tracking parameters are stripped before two addresses are compared, so one page arriving with a dozen campaign tags is one page and not a dozen. Subdomains of the same registrable domain count as part of the site, which keeps a locale or shop subdomain from being mistaken for somebody else's website. The scheme, the www variant, the trailing slash and the letter case are then compared across everything really fetched.
Crawler or scanner: which one you want
The crawler answers where can this site be walked, and how much of it did we reach. The scanner answers what is this site made of — the resources, images, cookies, trackers and technologies its pages pull in. One maps the territory, the other inventories what stands on it. Most people who arrive here wanting a full picture end up running both, in that order.
What the crawl reports about reach
Each of these is a real check, and each one speaks to whether your pages can be found and read at all.
Navigation coherence
Compares the navigation of every crawled page and checks that key pages are linked from it.
XML sitemap
An XML sitemap is published and readable.
robots.txt
A robots.txt file is published.
HTML sitemap (visitors)
Looks for a human-readable sitemap page. This is NOT sitemap.xml, which is audited by the SEO checks.
Indexability
Analysed pages are not blocked from indexing by a robots meta tag.
URL consistency
Compares scheme, www variant, trailing slash, case and parameters across the URLs really fetched.
Error responses
Crawled pages answered without a server error.
Performance measurement coverage
How much of the resource inventory could really be measured under the scan budget.
The crawl respects robots.txt and stops at your plan's page budget, so it reads a defined part of your site rather than all of it, and says which part. It never signs in, never solves a CAPTCHA and never works around bot protection: a page needing a real session is reported as not analysed, with the reason.
Enter your website address and VeriFixScan analyses it straight away. No account, no installation, nothing to add to your site.
Frequently asked questions
- What is a website crawler?
- A program that starts at one address, reads the page, follows the links and sitemap entries it finds, and repeats. It is how a search engine discovers what a site contains, and how you find out which of your pages are actually reachable.
- How do I crawl my own website?
- Submit your address and the crawl starts there. No file to upload, nothing to install: the pages are read over HTTP exactly as they are served to anyone else.
- Can it scan all the pages of a website?
- It discovers what is reachable from your entry address, your sitemaps, your robots.txt and your navigation, within the page budget of your plan. A page linked from nowhere and absent from every sitemap cannot be discovered by any crawler, ours included — and the coverage figures say how far the crawl actually got.
- Does the crawler follow sitemaps and robots.txt?
- Both, and for different reasons. Sitemaps and robots.txt declarations are read as sources of URLs; the robots.txt rules are then obeyed, so a disallowed path is recorded as blocked instead of being fetched.
- Does it crawl subdomains?
- Subdomains of the same registrable domain are treated as part of your site, so a shop or locale subdomain is followed rather than dropped as external. Unrelated domains are never crawled.
- Does it crawl other people's websites?
- It crawls the address you submit. Links pointing at other domains are verified as links, so a dead one is reported, but those pages are not crawled and not analysed.
- What happens when a page cannot be read?
- The crawl moves on immediately and records why: robots disallowed, login required, bot protection, a 403, a 429, a timeout, a network error or an off-site redirect. The rest of the analysis is completed and reported with those pages listed.
- What is the difference between crawling and scanning?
- Crawling discovers and reaches pages; scanning inventories what those pages contain. This page covers the first, the website scanner covers the second, and a full audit runs both against the same crawl.
- Can I crawl my website for free?
- The free scan crawls a sample of your site without an account. A larger page budget, and therefore a deeper crawl, comes with a plan.