Taxonomy overview ordering (drag and drop) does not work if the user does not have access to at least one item in the list.

core/modules/taxonomy/src/Form/OverviewTerms.php:

1. We checked vocabulary has "access to update tree"

  $update_tree_access = AccessResult::allowedIf(empty($pending_term_ids) && $vocabulary_hierarchy !== VocabularyInterface::HIERARCHY_MULTIPLE);  <strong>TRUE</strong>

2. The in loop for each term:
foreach ($current_page as $key => $term)

3. We check term "update access" and "update tree access" by this term.

  $update_access = $term->access('update', NULL, TRUE); <strong>FALSE</strong>
  $update_tree_access = $update_tree_access->andIf($update_access); <strong>FALSE</strong>

4. We rewrite $update_tree_access on previous step, so in the next loop iteration with update_access = true we will have update_tree_access = false

It means if user don't have right for at least one item in the vocabulary (entity_access or i.e. domain_access) he will not able to sort the list. And form will be broken.