Default nginx template returns 404 for WordPress's virtual robots.txt

The nginx config Ploi generates for a site contains:

location = /robots.txt  { access_log off; log_not_found off; }

With no try_files, nginx only ever looks for a physical file. On any WordPress site without one — which is the default, since WordPress generates robots.txt dynamically via the do_robots hook, as do Yoast, AIOSEO and Rank Math — /robots.txt returns 404. Crawlers never see the site's directives or its Sitemap: declaration.

This is a silent SEO defect: nothing in the panel indicates anything is wrong, and the site otherwise works perfectly. Google treats a 404 robots.txt as "crawl everything", so it can go unnoticed for months.

Fix:

location = /robots.txt  { try_files $uri /index.php?$args; access_log off; log_not_found off; }

That serves a physical robots.txt when one exists, and otherwise lets WordPress generate it. No downside for non-WordPress sites.

Please also apply it to the default webserver template so newly created sites aren't affected.

The same defect was reported and fixed in HestiaCP (https://github.com/hestiacp/hestiacp/issues/3729). It originates from the widely copy-pasted legacy nginx WordPress recipe.

Default nginx template returns 404 for WordPress's virtual robots.txt

1 total vote
Quick Actions
Activity
View recent activity and updates
Use arrow keys to navigate