Website crawler vs website scanner

A crawler is a discovery process. Given a starting URL, it fetches the page, extracts the links, and repeats — subject to a boundary (which host, which paths), a budget (how many pages, how deep), and the site's own rules in robots.txt.

Its output is a map: the URLs found, how each was reached, what status each returned, and which ones it declined to fetch. That last part is the one most often discarded, and the most useful — 'not crawled because robots.txt disallows /admin' and 'not crawled because no link points there' are different facts about a site.

A crawler alone finds structural defects, because structure is what it sees: links to pages that 404, redirect chains, orphan pages in the sitemap, and sections unreachable from any link.

A scanner is an analysis process. Given a page — its HTML, headers, and the resources it loads — it applies a set of checks and reports which fail, with the evidence.

A scanner needs the page to be handed to it. Run on a single URL it analyses that URL properly and says nothing about the rest of the site. This is why a report from a one-page tester is accurate and, on its own, misleading about scope.

Its output is a set of findings: this response has no Content-Security-Policy header, this image has no alt attribute, this title is duplicated, this canonical points elsewhere.

Because the two are usually combined, a report mixes two kinds of statement, and they have different reliability.

Coverage statements come from the crawler: 'we examined 412 pages'. They are only as good as the boundary and the budget. If the crawl stopped at 500 pages on a 5,000-page site, every percentage in the report describes 10% of the site.

Finding statements come from the scanner: '17 pages have no meta description'. These are reliable about the pages examined and say nothing about the pages that were not.

The practical consequence: 'no broken links were found' means 'no broken links among the URLs we reached'. A report that does not tell you what it did not reach cannot support that sentence, and you should read its absence of findings as an absence of evidence.

Product pages rarely use these words consistently, so it is more reliable to ask what the tool reports than what it calls itself. Four questions settle it.

Does it ask for one URL or for a domain? A single-URL field is a scanner; a domain field with a page limit means a crawl is going to happen first.

Does the result say how many pages were examined? If not, you cannot tell whether a clean report covered the whole site or its first twenty pages.

Does it say what it skipped, and why? Disallowed by robots.txt, outside the boundary, beyond the budget, never linked to — a crawler that tracks these can tell you; one that does not will silently present partial coverage as complete.

Does each finding come with evidence you can re-check yourself — a URL, a status code, a header, an element? That is the scanner half doing its job. A finding you cannot verify is one you will end up arguing about instead of fixing.

'Scanner' also means something else in security, where a vulnerability scanner probes a system for known weaknesses, often by sending crafted requests. That is an active test, usually requiring authorisation, and it is not what a website scanner in the auditing sense does.

Search engines describe their own discovery process as crawling and the interpretation step as indexing. The crawler/scanner split above is the same idea applied to auditing: find, then examine.

VeriFixScan does both, and keeps them separate in the result. The crawl reports which URLs were discovered, which were fetched, and why any were not. The scan reports the findings per page with their evidence.

That separation is deliberate: it is what lets you tell 'the site has no broken links' from 'the crawl did not get far enough to know'.