Before you edit robots txt in WordPress, look at what is already there. Visit /robots.txt on your domain and you will almost certainly see a file, even though nothing exists in your site’s root folder. WordPress generates it on request. Knowing where that output comes from, and what overrides it, decides which of the three editing routes you should take.
What WordPress puts in robots.txt by default
The file is produced by a core function, do_robots(). According to the WordPress developer reference for do_robots(), the default output is a User-agent: * line, a Disallow: rule for the admin path and an Allow: rule for the admin-ajax.php path. On a standard install it looks like this:
The admin area is blocked because it has no search value. The exception for admin-ajax.php exists because themes and plugins call it from the front end, and a crawler rendering the page may need it. Since WordPress 5.5, core also adds its own sitemap to the file; our guide to the WordPress sitemap covers what that sitemap includes and when an SEO plugin replaces it.
The virtual file and the physical file
The generated file is often called the “virtual” robots.txt, because it only exists when someone requests it. WordPress serves it when there is no real file at that address. Upload a physical robots.txt to the site root and the web server returns that file instead, so WordPress never runs do_robots() at all.
That override is the source of most confusion. Someone adds a physical file during a migration, forgets about it, and later wonders why the sitemap line their plugin promised never appears. If your edits are not showing up, check for a real file first.
Three ways to edit robots txt in WordPress
| Method | How it works | Watch out for |
|---|---|---|
| SEO plugin editor | The plugin changes the generated output from inside the dashboard | Ignored if a physical file exists in the root |
| Physical file via FTP or file manager | A real robots.txt in the root overrides the generated one | Core and plugin additions, such as the sitemap line, no longer appear unless you add them |
The robots_txt filter | Code in a theme or plugin modifies the output of do_robots() | Only runs while WordPress is generating the file |
| Do nothing | The core default stays in place | Fine for most small sites |
For most site owners the plugin editor is the right choice, because it keeps the generated file and its automatic additions. A physical file suits people who want the rules fixed and visible in version control. The filter is for developers who need the output to change with conditions.
The developer reference documents the hook as apply_filters( 'robots_txt', $output, $public ). The second argument tells you whether the site is set to be visible to search engines. A minimal example that appends one rule:
The “Discourage search engines” setting changed in 5.3
Older guides say that ticking “Discourage search engines from indexing this site” under Settings, Reading writes Disallow: / into robots.txt. That stopped being true in WordPress 5.3. The changelog in the developer reference reads: “Remove the ‘Disallow: /’ output if search engine visibility is discouraged in favor of robots meta HTML tag.”
This matters because a robots meta tag only works if crawlers can fetch the page and read it. A blanket disallow would have hidden that tag from them. If you inherit a site that still carries Disallow: /, it is coming from a physical file or a plugin, not from core. Our post on the meta robots tag explains how the tag-based approach works.
Rules that cost WordPress sites traffic
The best robots txt for WordPress is close to the default. Most of the long rule sets copied from forums do more harm than good. The ones to avoid:
- Blocking
/wp-content/or/wp-includes/. These hold theme CSS, JavaScript and images. Google renders pages, and if it cannot fetch those files it sees a different page from your visitors. - Using disallow to remove pages from search. A blocked URL can still be indexed from links, and crawlers can no longer see a noindex on it. Our post on indexed though blocked by robots.txt covers the fallout.
- A leftover
Disallow: /from a staging site, usually in a physical file that survived the move to production. - Over-broad patterns, such as a rule meant for tag archives that catches every path starting with the same letters.
Reasonable additions are narrow: internal search results, parameter URLs created by filters, and any AI crawler you have decided to opt out of. That last one is a business decision rather than an SEO one; see our guide to blocking AI crawlers.
How to add robots.txt to WordPress and check it
If you want to add robots.txt to WordPress as a real file, create a plain text file called robots.txt, paste in the default rules plus any narrow additions, and include your sitemap line yourself, since core will no longer add it. Upload it to the site root, which on most installs is the folder that holds wp-config.php, using your host’s file manager or an FTP client. Then load /robots.txt in a browser to confirm the server is returning your version rather than the generated one. If you run a caching plugin or a CDN, purge it first, or you may be looking at a stale copy of the old file.
After any change, check the robots.txt report in Search Console, which shows the version Google last fetched. Then inspect a few important URLs to confirm they are still allowed. Do this on the day of the change, not when traffic drops.
One limitation is worth stating plainly. Robots.txt is a request that well-behaved crawlers honour; it is not access control. Scrapers can ignore it, and the file is public, so listing a private path in it advertises that the path exists. Anything sensitive needs authentication, not a disallow line.
The practical rule for WordPress is short. Leave the default in place unless you have a specific crawling problem. If you edit, prefer the plugin editor or the filter so core keeps adding the sitemap. If you use a physical file, own it and review it after every migration. Our broader guide to robots.txt for SEO covers syntax and the crawl-versus-index distinction in more depth, and the strategy archive collects the related technical posts.
Frequently asked questions
Where is the robots.txt file in WordPress?
Usually nowhere on disk. WordPress generates it on request through do_robots() when no physical file exists. If a real robots.txt sits in the site root, the server returns that instead.
Does “Discourage search engines” block the site in robots.txt?
Not since WordPress 5.3. Core removed the “Disallow: /” output in favour of a robots meta HTML tag, so crawlers can still read the page and see the instruction.
Should I block wp-content in robots.txt?
No. It holds the CSS, JavaScript and images Google needs to render your pages. Blocking it can make Google see a broken version of the site.
Why are my plugin’s robots.txt changes not showing?
Most likely a physical robots.txt in the root is overriding the generated file. Remove it or edit it directly.
The takeaway WordPress serves a sensible robots.txt by default. Edit it through a plugin or the robots_txt filter when you need to, and treat a physical file as something you own and must maintain.
