- Issue created by @prudloff
This file does not include more recent fixes like #3123285: Actually exclude user register, login, logout, and password pages from search results in robots.txt (current rules are broken) → , #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO → or 📌 Update robots.txt with all README files core ships with RTBC .
Don't allow core to scaffold it robots.txt file.
Install the robotstxt module.
The default config includes an outdated robots.txt file compared to core.
Backport the fixes to the module.
Another solution would be to use web/core/assets/scaffold/files/robots.txt directly.
Active
1.0
Code