Have you tried installing domain and robotstxt modules?

Problem/Motivation

I have tried to customize robots.txt using domain and robotstxt module without susceed. My use case is specific to pantheon that blocks drupal robots.txt by default in order to remove multidevs testing environments from crawl.

Steps to reproduce

On a pantheon Drupal 10.3 installation install robotstxt and domain modules using composer. Configure your domains and enable robots using per domain configuration on /admin/config/search/robotstxt

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes