Default nginx template returns 404 for WordPress's virtual robots.txt
The nginx config Ploi generates for a site contains:
location = /robots.txt { access_log off; log_not_found off; }
With no try_files, nginx only ever looks for a physical file. On any WordPress site without one — which is the default, since WordPress generates robots.txt dynamically via the do_robots hook, as do Yoast, AIOSEO and Rank Math — /robots.txt returns 404. Crawlers never see the site's directives or its Sitemap: declaration.
This is a silent SEO defect: nothing in the panel indicates anything is wrong, and the site otherwise works perfectly. Google treats a 404 robots.txt as "crawl everything", so it can go unnoticed for months.
Fix:
location = /robots.txt { try_files $uri /index.php?$args; access_log off; log_not_found off; }
That serves a physical robots.txt when one exists, and otherwise lets WordPress generate it. No downside for non-WordPress sites.
Please also apply it to the default webserver template so newly created sites aren't affected.
The same defect was reported and fixed in HestiaCP (https://github.com/hestiacp/hestiacp/issues/3729). It originates from the widely copy-pasted legacy nginx WordPress recipe.
-
Dennis moved item to board Live
2 days ago -
Dennis moved item to board In progress
2 days ago -
Filipe Seabra moved item to project Bugs
2 days ago -
Filipe Seabra created the item
2 days ago