Views entity_reference filter incompatible with TermSelection plugin

Problem/Motivation

After upgrading to 10.4 and installing https://www.drupal.org/project/views_core_entity_reference → I was trying to track down why the ordering in my taxonomy_term entity reference select list filters weren't matching what we had on 10.3.

I tracked it down to this line which short-circuits the custom Taxonomy based querying if a $match or $limit is passed.

In the previous iteration of this filter, there was no $limit parameter passed, but the one that was eventually committed does.

Steps to reproduce

Install 10.4
Install views_core_entity_reference
Add a Term reference field for a Vocabulary using the Select list widget
Add terms to the Vocabulary and reorder them to a non-alphabetical weighting
Notice the terms are sorted alphabetically in the filter.

Proposed resolution

Not too sure, I believe the $limit there is to reduce performance overheads of loading a massive list of options.

Remaining tasks

Figure out a fix...