Removing trailing slashes from robots.txt

Landed here from Bug smash initiative.

+1 Patch is applied successfully for version 11.x-dev too and starting test for 11.x-dev version

Tests are failing for #8. @abhaypai could you perhaps turn this into a working MR against 11.x?

Prashant.c → made their first commit to this issue’s fork.

@Anybody
Created MR against 11.x by taking the changes from #8 🐛 Removing trailing slashes from robots.txt Needs review .

Thank you.

Thanks @Prashant.c! All tests are passing green, so I think we should get this out of the way! Marking this RTBC.

According to the Google robots.txt description here: https://developers.google.com/search/docs/crawling-indexing/robots/robot... , the /search should match any path that starts with /search. So I do not think we need both /search and /search/ (and similar for the second change). The rules from robots.txt should be the "starts with" rules.

The second question is, we have /admin/ in robots.txt. /admin path is valid as well, so why we are not changing this too?

Can it be documented in the issue summary what paths were chosen and why.