Validation is called when terms are selected in tree.

When clicking any term name from taxonomy tree, the form rebuild is ran.
All validation handlers are called even if they are not necessary, considering that we are just switching the term that is viewed.

As an example consider that we have a required field which is not completed (data being populated with some import scripts). Once we entered a term we cannot switch to another, until that required field is not empty.

Update1:
Same issue when you are trying to search another term and you already have a term opened with a required field that is empty.