Disallow crawling paths under /node by default in robots.txt

Add Workaround in Issue Summary.

Although Google recommends descriptive URLs, there is nothing "wrong" with /node paths.

Because using Redirect to maintain canonical URLs → , that is, <link rel="canonical" />, works, I wonder whether it is ideal not to index /node paths by default.

These are just my first thoughts after a few minutes' consideration.

Although Google recommends descriptive URLs, there is nothing "wrong" with /node paths.

That's totally true, but not the focus here. I outlined the problem in the issue Summary:

Getting paths such as /node/100 indexed instead of the human readable URL alias /my-alias is bad for SEO ...

The point is, that in most cases, you want the human readable path indexed in the first place, not the node/NID path -- even if it gets redirected.

I have updated the Issue Summary to make this point clearer.

Also note:

If you have multiple pages that have the same information, try setting up a redirect from non-preferred URLs to a URL that best represents that information. If you can't redirect, use the rel="canonical" link element instead. But again, don't worry too much about this; search engines can generally figure this out for you on their own most of the time.

https://developers.google.com/search/docs/fundamentals/seo-starter-guide...

Sure, and the Redirect module can take care of that, as far as I see, should a /node/100 path get exposed and indexed by mistake ...

Or do you have another point with sharing that sentence?

Again, the aim with this MR is to get the correct alias indexed in the the first shot, by blocking /node/100 from getting indexed in the first place.

In my experiences that article is correct: search engines respect Core's rel="canonical", with or without Redirect. I am trying to understand the downside of having a /node URL indexed to which later the author adds a path alias, which the search engines then accept.

Does the opposite ever happen?

I don't know, I created this issue because /node/100 paths got indexed, for some reason.

But what would be the downside to doing this?

I don't know, really. It just seems a strong default not to index /node. It's a bit of a singular case but this very website would be largely un-indexed with that default. 🙁

Actually that's not completely true because of the /issues auto path.

I do appreciate getting the tires of the MR kicked, don't get me wrong!

I just think that in the majority of new installations, you do not want node/100 paths indexed. And if that's true, we should make the preferred behaviour the default, non?

Sorry if I'm just repeating something. But I'm thinking of a novice site builder (mom and pop shop). If you don't have pathauto and don't manually set the alias then the URL will be node/123. Think we can all agree that's bad practice and standards. But idk how to vote for this one lol. Robots.txt should be part of core maybe?

Heh, there has been some debate, but I think the gist of it was condensed into the last sentence of comment #16 -- that the majority would benefit from this.

Also, this change would probably not cause any big problems, but rather a theoretical challenge, for a select few.