A robots.txt unreachable error means Google asked for your robots.txt file and got a server error or no answer at all. Most site owners file it alongside the minor Search Console notices. That is the wrong instinct. Robots.txt is the first thing Google fetches before it crawls anything else on a host, so when that request fails, Google does not know what it is allowed to crawl, and it errs on the side of crawling nothing.
Why robots.txt unreachable is worse than no robots.txt
The counter-intuitive part is that having no file is fine, while having a broken one is not. Google’s robots.txt specification spells out the difference. For client errors: “Google’s crawlers treat all 4xx errors, except 429, as if a valid robots.txt file didn’t exist.” No file, no rules, and Google crawls everything.
Server errors get the opposite treatment. “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file. If Google can’t fetch a new version, for the next 30 days Google will use the last good version, while still trying to fetch a new version.” After that, what happens depends on the site’s general availability.
Gary Illyes of Google put the risk more bluntly, as reported by Search Engine Roundtable: “A robots.txt file that returns a 500/503 HTTP status code for an extended period of time will remove your site from search results, even if the rest of the site is accessible.” He added: “Same goes for network timeouts.”
How Google handles each response
The status code your server returns for /robots.txt decides everything. This table summarises the rules from Google’s specification.
| Response to /robots.txt | What Google does | Risk |
|---|---|---|
| 200 with valid rules | Follows the rules; caches the file “for up to 24 hours” | None |
| 4xx (except 429) | Treats it “as if a valid robots.txt file didn’t exist” | None; everything is crawlable |
| 5xx, first 12 hours | “Google stops crawling the site but keeps trying to fetch the robots.txt file” | High |
| 5xx, next 30 days | Uses “the last good version, while still trying to fetch a new version” | Rising |
| Network timeout | Same as a 5xx, per Illyes: “Same goes for network timeouts” | High |
| File over 500 KiB | Content after the limit is ignored | Rules silently dropped |
Two consequences follow. First, the 12-hour window is short: an error that starts on a Friday evening can halt crawling before anyone looks. Second, the 30-day fallback to the last good version is a safety net, not a fix. If your rules changed in that time, Google is still working from the old ones.
What usually causes it
The file itself is almost never the problem. A plain text file does not break. What breaks is the path between Googlebot and that file. The common causes are:
- Firewall or CDN bot rules. A security setting that challenges or blocks automated traffic can return an error page, or nothing, to Googlebot while browsers see the file normally.
- Server overload. When the origin is struggling, robots.txt fails along with everything else, and it is the request that matters most.
- Timeouts. A slow origin that does not answer in time counts as unreachable, even if it would eventually serve a 200.
- Misconfigured hosting. Rewrite rules that route
/robots.txtthrough the application, a plugin that generates the file dynamically and throws an error, or a hostname that is not configured on the server.
The firewall case is the one that catches experienced teams, because every manual check passes. You open https://example.com/robots.txt in a browser, see the rules, and conclude the report is wrong. Googlebot, coming from different IP ranges with a crawler user agent, may be getting a challenge page or a 503. If you have not read our guide to server errors (5xx), the same diagnosis applies here, narrowed to a single URL.
How to fix a robots.txt unreachable error
- Check what Googlebot sees, not what you see. Use URL Inspection on the robots.txt URL, or the robots.txt report in Search Console, and look at the fetch status and time.
- Read the server and CDN logs for requests to
/robots.txtwith a Googlebot user agent. Look for 5xx codes, timeouts and challenge responses. - Exempt verified crawlers from bot rules. Make sure firewall rules allow verified search engine crawlers through, and that
/robots.txtis never served a challenge. - Serve the file statically. A plain file served directly, or cached at the edge, survives application errors that would take a dynamic one down.
- Fix the capacity problem. If the origin is overloaded, robots.txt is only the first casualty. Look at crawl budget as well, because Google reduces crawling on hosts that respond slowly or with errors.
- Request a recrawl. Once the file returns a fast 200, ask Search Console to refetch it. Google caches robots.txt for up to 24 hours, so expect a short lag either way.
While you are there, check the rules themselves. An error that has just been fixed is a good moment to confirm nothing important is disallowed; our guide to robots.txt for SEO covers what the file should and should not contain, and blocked by robots.txt covers the separate case where the file is reachable but too strict.
Monitoring so it does not happen silently
Search Console tells you after the fact. An uptime monitor pointed specifically at /robots.txt, checking for a 200 within a sensible response time, tells you within minutes. That matters because of the 12-hour rule: an outage you catch in ten minutes never stops crawling, while one you find in Monday’s report may already have.
Add a second check after any change to the CDN, firewall or hosting. Those are the moments the problem is introduced, and they are rarely tested from a crawler’s point of view.
A limitation worth stating: Google’s documentation describes the rules, not the exact thresholds for “an extended period of time”. Neither the specification nor Illyes gives a number of days after which a site leaves the results, and what happens after the 30-day fallback “depends on the site’s general availability”. You cannot calculate how long you have, which is the argument for fixing it the same day rather than waiting to see.
The practical order of priority is simple. A robots.txt error outranks almost every other item in the indexing reports, because it affects every URL on the host at once. Fix it first, confirm Googlebot gets a fast 200, then return to the page-level issues.
Frequently asked questions
Is it better to delete robots.txt than have it error?
Yes, in the narrow sense that a 404 is treated as if no file existed and Google crawls normally. A 5xx stops crawling. The right fix is a file that returns a fast 200.
How long before an unreachable robots.txt hurts rankings?
Google stops crawling in the first 12 hours of 5xx errors, then uses the last good version for 30 days. Google has not published an exact point at which pages are removed.
Why does the file load in my browser but Google says unreachable?
Usually a firewall or CDN rule treating Googlebot differently, or intermittent timeouts. Check your logs for Googlebot requests to the file.
Does a 429 on robots.txt count as unreachable?
Google excludes 429 from the 4xx rule, so it is not treated as a missing file. Treat rate limiting of Googlebot on robots.txt as an error to fix.
The takeaway A missing robots.txt is fine; an unreachable one halts crawling. Check what Googlebot receives, exempt it from bot rules, serve the file statically and monitor it directly.
