Implemented a single authoritative robots policy for aguiarinjurylawyers.com on May 12, 2026.
Shipped:
- Deployed a physical root robots.txt at /var/www/html/public_html/robots.txt and local source at /Users/samaguiar/Documents/Projects/robots.txt.
- Updated /Users/samaguiar/Documents/Projects/wp-content/mu-plugins/seo-sitemap-authority.php so the fallback WordPress robots output returns one User-agent: * block and exactly one Sitemap directive.
- Included only intentional disallows: /wp-admin/, /wp-login.php, /?s=, /wp-content/uploads/gravity_forms/, and /wp-content/advanced-headers-test.php.
- Preserved Allow: /wp-admin/admin-ajax.php so rendering/AJAX assets are not overblocked.
- Confirmed the canonical sitemap remains https://aguiarinjurylawyers.com/sitemap_index.xml.
- Verified CSS and JS URLs are allowed by parser and return HTTP 200 publicly.
- Verified /wp-content/advanced-headers-test.php returns HTTP 403 publicly.
- Added a narrow .htaccess robots cache override block locally and on production, with remote backup at /var/www/html/public_html/.htaccess.bak-robots-cache-20260512-1730.
QA evidence:
- php -l passed for seo-sitemap-authority.php locally and on production.
- WP-CLI origin output returns the new one-block robots policy.
- Cache-busted public /robots.txt returns the new root robots file.
- Python robotparser checks passed for key paths: homepage allowed, admin/login/search/gravity_forms/advanced-headers-test blocked, admin-ajax/CSS/JS/uploads allowed, one sitemap detected.
- Sitemap index returns 200; /wp-sitemap.xml and /sitemap.xml still 301 to the canonical Rank Math sitemap index.
Challenges:
- Plain https://aguiarinjurylawyers.com/robots.txt still serves the old cached body with cf-cache-status HIT, age about 234k seconds, last-modified May 10. The new policy is live at origin and cache-busted URLs, but the old edge object needs Cloudways Autonomous purge or natural cache expiry.
- Cloudways API purge using the local vault credentials returned HTTP 403.
- Standalone Cloudflare purge returned success but did not affect production edge cache, matching prior memory that this zone is not the serving edge for Cloudways Autonomous.