Glossary
Retry-After
Retry-After is a response header telling the client how long to wait before repeating a request. It accepts either a number of seconds or an HTTP date, and it appears mainly on two status codes: 429, where a client has exceeded a quota, and 503, where the service is temporarily unavailable. Its value is that it replaces a guess with an instruction, which is the difference between a client that waits correctly and one that retries into a wall.
The two formats
A delay in seconds, which is relative and immune to clock differences between the client and the server.
An HTTP date, which is absolute and useful when the resumption time is genuinely known — a scheduled maintenance window ending at a fixed hour.
Clients are required to accept both, so the choice is about which one the server can state accurately.
A delay in seconds is the safer default, because an absolute date is wrong for every client whose clock is off.
On 503, where it changes crawler behaviour
A 503 with Retry-After tells a crawler the unavailability is temporary and roughly how long it will last.
Search documentation is explicit that a 503 during maintenance is handled far better than a 200 serving a maintenance page, because the status is what conveys the state.
A maintenance page returning 200 asks the crawler to index the maintenance notice, which is how sites end up with a maintenance message in search results.
The pairing that works is a 503 status, a Retry-After, and a human-readable page in the body for anyone who arrives during the window.
On 429, where it changes client behaviour
A client that reads it waits the stated period and succeeds on the next attempt.
A client that does not read it backs off by doubling, which usually waits far longer than necessary.
A client that does neither retries immediately, which keeps the quota exhausted and turns one client's mistake into sustained load.
Which is why it is worth sending even when the window is short: the short window is exactly the case where guessing is most wasteful.
How it is observed
Responses received during a scan are read for it, alongside the rate-limit budget headers.
A maintenance page is detected separately, and a maintenance page served with a 200 is reported as the status mismatch it is.
Nothing is sent to provoke a 429 or a 503, so the header is only seen where those responses occurred naturally.
Absence is reported as absence, since a server may send it under conditions the scan did not produce.
Where it is easy to get wrong
Sending an absolute date computed from the server clock, which is wrong for every client whose clock differs.
Sending a value far larger than the real window, which makes well-behaved clients wait for no reason.
Sending it on a status code that is not a refusal, where it has no defined meaning and is ignored.
Omitting it on a scheduled maintenance window, which is the one case where an absolute date is genuinely the better format.
Frequently asked questions
- Seconds or a date?
- Seconds, unless you genuinely know the absolute resumption time. A date is wrong for every client whose clock disagrees with yours.
- Does it matter during planned maintenance?
- Yes. A 503 with Retry-After tells crawlers the state is temporary. A maintenance page returning 200 asks them to index the maintenance notice instead.
- Will clients actually honour it?
- Search crawlers do. Arbitrary clients vary, and a client that ignores it is no worse off than one that never received it — so sending it costs nothing.
Sources
Related
VeriFixScan crawls a site and applies its checks to every page it reaches, keeping the evidence behind each finding. Scanning one website is free.
Scan a website