Sort values by weight if referencing a Taxonomy

This looks like a good idea, but I wonder if it could be made more generic? Such as providing an option to sort by weight whenever there is a weight field, or any other field on the referenced entity.

Alternatively, being able to sort by another view could take handle most use cases. For example, we're using draggable views → to arrange referenced entities in an arbitrary order. This works with both lists and entity reference fields but not Views Entity Reference Filters.