Passing referenced entity in Twig url function references parent entity instead of referenced entity

Problem/Motivation

I have a taxonomy term referencing a language config entity where I want to generate a URL using the referenced language. When I pass this language reference in the url Twig function, it appears that it's trying to access the parent term instead of the referenced entity. If I create the variable in a preprocess hook, this issue does not appear. It seems to be something about doing it in Twig. I don't think this is limited to my specific use case but I've provided steps matching my use case.

Steps to reproduce

1. Create an vocabulary
2. Add a custom entity reference field ("Other", "Language" config entity)
3. Override the bundle Twig template
4. In the overridden template, try to out put the URL, passing the language entity
5. Note the error thrown, accessing an undefined method getId() on Term object

{{ url('<front>', {}, {'language': term.field_language.entity}) }}

I haven't been able to figure out where the entities get mixed up. Outputting the referenced entity with dpm works as expected, as does just outputting that field value. I've only seen the issue when referencing it in this way.