To answer @cilefen
The proposal above seems analogous to Drupal's blocking of Google FLoC
https://www.drupal.org/project/drupal/issues/3209628 →
FLoC has now gone away and so have the blocking headers.
https://www.drupal.org/project/drupal/issues/3260401 →
That said, my preference is that blocking AI bots shouldn't be in core;
it should be in contrib for those who want to opt out.
Perhaps I am still on a sugar high from drinking the AI Kool-Aid but I think
the benefits of LLMs will greatly outweigh the risks.
AI bots reading content should be considered transformative use.
Imagine multiple super intelligences reading the medical literature,
coming up with new treatments and models -- but wait -- a content troll company
like Elsevier prevents this with exorbitant fees or a complete blockade.
For your consideration, an essay from Benedict Evans:
Generative AI and intellectual property
https://www.ben-evans.com/benedictevans/2023/8/27/generative-ai-ad-intel...