Removing trailing slashes from robots.txt

Landed here from Bug smash initiative.

+1 Patch is applied successfully for version 11.x-dev too and starting test for 11.x-dev version

Tests are failing for #8. @abhaypai could you perhaps turn this into a working MR against 11.x?

Prashant.c → made their first commit to this issue’s fork.

@Anybody
Created MR against 11.x by taking the changes from #8 🐛 Removing trailing slashes from robots.txt Needs review .

Thank you.

Thanks @Prashant.c! All tests are passing green, so I think we should get this out of the way! Marking this RTBC.

According to the Google robots.txt description here: https://developers.google.com/search/docs/crawling-indexing/robots/robot... , the /search should match any path that starts with /search. So I do not think we need both /search and /search/ (and similar for the second change). The rules from robots.txt should be the "starts with" rules.

The second question is, we have /admin/ in robots.txt. /admin path is valid as well, so why we are not changing this too?

Can it be documented in the issue summary what paths were chosen and why.

I agree @poker10, we should remove all trailing slashes, and I have updated the MR to reflect this.

I have added the table below in the Issue Summary, is that sufficient documentation @smustgrave, or do we need some more?

From How Google interprets the robots.txt specification > URL matching based on path values:

PS. Personally, I would add Disallow: /node, since:

1. The vast majority of sites use Pathauto, installs January 2025:
   
   

     Drupal core:	723,408
     Pathauto:	514,780

   From https://www.drupal.org/project/usage →

2. Getting paths such as /node/100 indexed instead of the human readable URL alias /my-alias is bad for SEO ...

... but that's for another issue :)

I created an issue about disallowing /node.

The table outlining the rules I wanted to add in the Issue Summary got lost, now it's actually added.