Media library remove button triggers field constraint validation

I also encountered this issue. Somehow the ConstraintValidator should be aware of which media items are removed. For now I implemented a dirty solution which checks if the remove button of a specific media item is clicked and skips the validation for that media item:

public function validate($value, Constraint $constraint) {
    $field_name = $this->context->getPropertyName();
    $triggering_element_name = $this->requestStack->getCurrentRequest()
      ->get('_triggering_element_name');

    foreach ($value as $index => $item) {
      $media = $item->entity;
      if (!$media instanceof MediaInterface) {
        continue;
      }

      if ($triggering_element_name === $field_name . '-' . $index . '-media-library-remove-button') {
        // Remove button was clicked, so skip validation.
        continue;
      }

      if (!$this->canUse($media, $value->getEntity())) {
        $this->context->addViolation($this->t('This media item is already being used.'));
      }
    }
  }

Update: The _triggering_element_name contains extra text when the constraint is validated on a field which is added to a paragraph type. In that case,

if ($triggering_element_name === $field_name . '-' . $index . '-media-library-remove-button') {

can be replaced with

if (stripos($triggering_element_name, $field_name . '-' . $index . '-media-library-remove-button') === 0) {

which will possibly lead to other issues, but works in my case.