Cannot get this to work in a NOT IN setup

Problem/Motivation

I would like to use the entity reference filter with the "is not one of" setup.
I receive an AJAX error on the views edit interface.
In the error log i see this message:
Error: Attempt to assign property "type" on null in Drupal\views\ManyToOneHelper->ensureMyTable() (line 254 of /.../web/core/modules/views/src/ManyToOneHelper.php).

Steps to reproduce

The entity reference filter works correctly if the filter is "is one of".
I use Drupal 11.