Integrate with the Dark Visitors API

Problem/Motivation

AI bots are scraping sites aggressively, and it's leading to downtime in some cases. Dark Visitors is an effort to document the user agents of those bots with some info about them.

Steps to reproduce

Without the RobotsTxt module, a site administrator has to visit the Dark Visitors documentation page, copy & paste the example robots.txt file, and add it to their codebase. With the RobotsTxt module, they still have to do the step of remembering to visit the site and copy & paste the example. Dark Visitors has an API that can automatically pull from the list and generate a robots.txt file.

Proposed resolution

That Robots.txt integrate from the API. They offer a WordPress plugin which does the same, so the RobotsTxt module developers could take a look at that for an example.

Remaining tasks

Build the feature.

User interface changes

It would require a setting to so that the integration is optional.

API changes

Unsure.

Data model changes

Unsure.