- Issue created by @hash6
This seems to be a task or a feature request rather than a bug. robots.txt is editable by site operators.
- 🇮🇳India anmolgoyal74
I think the CSS, JS and Images are allowed to be crawled. In D7, the modules, themes, profiles and misc directories are not not allowed to be crawled that's why it was add in robots.txt of D7. I think we can close this.
https://git.drupalcode.org/project/drupal/-/blob/7543eae758df91a217e6464...
- First commit to issue fork.
- 🇺🇸United States smustgrave
Think we need some research first. Anyone check the history and see why these were added? Also this seems like something a project can do themselves. There's also https://www.drupal.org/project/robotstxt →
- Status changed to Closed: cannot reproduce
14 days ago 9:59pm 27 February 2025 - 🇸🇰Slovakia poker10
robots.txt does not seems to block
/modulesor/themesdirectories, see: https://git.drupalcode.org/project/drupal/-/blob/11.x/robots.txt?ref_typ... . This is different from D7 behavior, which blocks these directories by default.I agree with #4 that there is probably nothing needed here. Feel free to reopen if you still think anything is needed, but will need an issues summary update in such case. Thanks!