Allowed Memory Size exhausted Issue on large data set

Problem/Motivation

I have Taxonomy A Referenced from Taxonomy X
Taxonomy X has more than 10000 records
When Try to Edit or Add Taxonomy A, memory exhaust issue occurs.

Steps to reproduce

Add Taxonomy X as Reference field in Taxonomy A
Import / Create more than 10000 Records in Taxonomy X
Go to Taxonomy A
Edit or Add New Taxonomy A term
You will encounter Allowed Memory Size exhausted Issue
Is there a way to overcome this issue? I don't want to use Autocomplete Delux module since comfortable with this UI

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes