Default language is not EN, XML sitemap seems to be not working

Problem/Motivation

I've installed the sitemap module and it has different issues - maybe due to my configuration?

On my site, I've installed the German language as default.
English is not being used (but can't be disabled).
So all content and everything is German only.

When I use my standard sitemap and switch it from English to German, I'll see a 404 for /sitemap.xml
When I delete this and create a new one in German, I'll see a 404 for /sitemap.xml
Only when I have two sitemaps configured under: /admin/config/search/xmlsitemap - one for German, one for English - then I can see the sitemap.xml

In the sitemap.xml is only the frontpage with /en at the end - where is the de frontpage? (/en will be redirected to / directly, since there is no en on my site).

Then I've added a taxonomy term to sitemap.
It has got around 10k terms, generation took longer, I can see the two folders (2 languages) in my directory:
/web/sites/default/files/xmlsitemap/
there are 3 files:
index.xml
1.xml
2.xml
Generation seems to be have been successful.
(the en Folder only contains 1.xml and this is much smaller)

Then I delete the cache, reload the sitemap.xml and I still see only the frontpage/en link.
Where is the link to the DE generated sitemap?
Do I do anything wrong?

Steps to reproduce

D10.2
1 language which is not EN
Install XML Sitemap
Configure accordingly.

PS:
I've seen:
https://www.drupal.org/project/xmlsitemap/issues/2810833 →
https://www.drupal.org/project/xmlsitemap/issues/3006297 →
but it did now answer my question ("just" solved the problem with 404 from before).