Glossary
Duplicate content
Duplicate content is substantially the same content reachable at more than one URL, whether within a site or across sites. There is no penalty for it in the ordinary case: search engines pick one URL as canonical and consolidate the signals onto it. The costs are real but indirect — the chosen URL may not be the one you wanted, inbound links are split until consolidation happens, and crawl effort is spent on copies.
Where it comes from inside a site
URL variants of the same page: with and without `www`, with and without a trailing slash, over HTTP and HTTPS, with tracking parameters appended.
Faceted navigation and sorting, where each combination of filters produces a URL with substantially the same products in a different order.
Printer-friendly and AMP-style alternate versions of the same article.
Pagination and session identifiers, which multiply a listing into many near-identical pages.
Product descriptions supplied by a manufacturer and used verbatim by every retailer, which is duplication across sites rather than within one.
Why there is no penalty, and what the cost is instead
Google's documentation states that duplicate content does not result in a penalty in the normal case, and that the system simply selects a canonical version.
The cost is that the selection may not match your intent. A parameterised variant chosen as canonical is the URL that appears in results.
Inbound links pointing at different variants are consolidated only once a canonical is chosen, and until then they are divided.
Crawl effort spent re-fetching copies is effort not spent on pages that changed, which matters on large sites.
Deliberate, large-scale duplication intended to manipulate rankings is a separate matter and is addressed by the spam policies.
How to consolidate
A `rel=canonical` tag on every variant pointing at the preferred URL. It is a strong hint, not a directive, and is normally respected when it is consistent.
301 redirects where only one version should be reachable at all, which is stronger than a canonical tag and removes the duplicate entirely.
Consistency in internal links, the sitemap and the canonical tags. A site that links to one form and declares another as canonical is sending contradictory signals.
`noindex` only where the duplicate should be removed rather than consolidated, since it discards the signals instead of transferring them.
How to find it
Compare the set of URLs that return 200 against the set declared canonical. Every URL that is canonical to something else is a duplicate by construction.
Look for identical or near-identical titles and meta descriptions across URLs, which is the cheapest available proxy for duplicated bodies.
Test the four host and protocol variants of the same path; if more than one returns 200, the duplication is structural.
Search Console's Page indexing report names URLs excluded as duplicates and states which URL was chosen instead, including when it differs from your declared canonical.
Frequently asked questions
- Is there a duplicate content penalty?
- No, in the ordinary case. Search engines select one canonical version and consolidate. Penalties apply to deliberate manipulation, which is a different thing.
- Does a canonical tag guarantee which URL is chosen?
- No. It is a strong signal that is usually respected, particularly when internal links and the sitemap agree with it.
- Should duplicates be noindexed?
- Only if you want them removed rather than consolidated. noindex discards the page's signals; a canonical tag transfers them.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website