This is a follow-up to: #3389611-17: Update robots.txt with all README files core ships with → .
We should decide whether to disallow the config sync directory README.txt by the robots.txt.
The file is generated in the sites/default/files/config_ZZZ/sync directory (see the SiteSettingsForm::createRandomConfigDirectory()).
A rule like sites/default/files/*/sync/README.txt could be considered, even though the wildcards are officially not supported.
The sites/default/files/*/sync/README.txt is not excluded by robots.txt.
Decide whether this should be disallowed by robots.txt. If yes, add it, if not, close the issue.
Closed: won't fix
11.0 🔥
Last updated
I think we should close this as a won't fix. How would a crawler find it. And if it did the .htaccess should deny access. And many sites put configuration out of the web root anyway.