Update robots.txt with all README files core ships with

Good find, thanks!

That said, there are also still a lot of README.txt files in various places. Search with:

[ayrton:maintainer | Mon 18:41:50] $ find ./ -name README.txt | grep -v "vendor" | grep -v "node_modules"

So, I think we need to add the README.md entry, but not remove the README.txt one.

We can have a followup to consider converting other READMEs.txt to .md.

Thanks!

Does /README.txt match all readme files? I think it matches only the one in the document root.

Whoops, you may be right. If so, among all the various READMEs, some are excluded by other rule (like the /core/ rule. The files that are still allowed to be indexed are:

.//modules/README.txt
.//themes/README.txt
.//composer/Metapackage/README.txt
.//composer/Plugin/VendorHardening/README.txt
.//composer/Template/README.txt
.//sites/README.txt

I tried reading http://www.robotstxt.org/robotstxt.html but it isn't really clear one way or another on this point. Not really clear from Google either.

Maybe someone can just quickly test it to confirm?

I tested those with the official Google Webmaster Tools robots.txt tool.

Those files are not blocked by Drupal's current robots.txt file.

Should we re-scope this one?

I found a better technical description of the robots.txt format at the w3c specification: https://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1

It is extremely limited. No wildcards or relative paths are supported. This is basically the full spec:

The "Disallow" field specifies a partial URI that is not to be visited. This can be a full path, or a partial path; any URI that starts with this value will not be retrieved. For example,

    Disallow: /help disallows both /help.html and /help/index.html, whereas
    Disallow: /help/ would disallow /help/index.html but allow /help.html. 

So we should probably just maintain a list of the README files that core ships with.

Added the remaining README files.

I think the list looks good, thanks!

I reviewed the files and there can be one more README.txt generated in the sites/default/files/config_ZZZ/sync directory (see the SiteSettingsForm::createRandomConfigDirectory()). But not sure if we can target also this file with something like this: sites/default/files/*/sync/README.txt.

The last submitted patch, 9: 3389611-9.patch, failed testing. View results →

@poker10, unfortunately we can't target dynamic paths in robots.txt and we wouldn't want to disclose the location of the config folder.

This patch also updates the copy of robots.txt intended for scaffolding.

If no wildcards are supported, how is this existing robots.txt entry supposed to work?

Disallow: /*/media/oembed

I proposed the suggestion with the config directory based on this existing entry. It was added by: #3271222: Include Disallow Oembed media links in the robots.txt file for better Drupal SEO →

Wildcards are supported in practice by major crawlers.

Huh, very interesting.

That said, I think @pfrenssen's approach is workable. No need to try "supported in practice" wildcards when the list is a fairly short and consistent one.

So in case there is no concern that we do not disallow the readme in the config directory (sites/default/files/*/sync/README.txt), then I think this could be RTBC.

@poker10 Maybe we could add a followup about that case? If the site is using private files, it should already be access denied, but a site might not be. There's a fairly strongly worded .htaccess in the config sync directory also. So I think any additional rule would be a hardening. It could be discussed in a separate issue.

So we need two followups:

1. One to discuss converting other READMEs to Markdown.
2. One to discuss whether we should add additional defense-in-depth around the config sync README,

Thanks everyone!

@xjm Yes, that sounds good, thanks.

I have created the follow-ups:

📌 Change all README.txt files to README.md Needs work
📌 Disallow the config sync directory README.txt by the robots.txt Active

I have also discovered, that we still have references to the old README.txt in the root directory on some places (after it was renamed), so created one more issue as well: 🐛 Change references to README.txt in root directory RTBC

As this changes a scaffolded file I think we should only do this in 11.x

Committed 505a646 and pushed to 11.x. Thanks!

Automatically closed - issue fixed for 2 weeks with no activity.

Manual testing was completed in #7. Therefore removing the tag.