Update the index logic and not to reindex whole dataset

Some of this is in #3316432-28: Timeout when processing large files → but we're missing a check to delete any records that are no longer valid.

I wonder how we should handle that.

There are two scenarios.

1) A record exists in a dataset, and then is removed - we should remove it from the index
2) A record exists in a dataset, is valid at some point, is indexed and then becomes invalid, should we remove it from the index or retain the previous valid one?

Scenario 1 is obvious what we should 2, scenario 2 is less clear cut.

Thoughts?

In 2, how did it become invalid?

e.g the source data at remote URL changed or user uploaded a new file on the dataset

My thoughts are different for community vs NSW use cases.

For community, I think the record should be deleted from the index if it becomes invalid. This gives you data integrity.

But based on HA strategy for NSW, we would normally prefer to provide stale content than a 'page not found' while the problem is resolved.

I think we go with the community solution in this case, and if we want to override it, we can look at that later as a hook or otherwise on our end.

We could make it configurable per dataset

Screenshot of how the field might look

Here's the changes against 🐛 Timeout when processing large files Fixed (in the do not test patch) as well as a combined patch

@larowlan Again great work! Only one comment.

@@ -304,6 +325,9 @@ class Dataset extends ContentEntityBase implements DatasetInterface {
       $queue->createItem(new ProcessingOperation(ProcessingOperationEnum::END, $destination_id));
+      if ($this->getInvalidValuesHandling() === self::INVALID_REMOVE && $invalid_deltas = $iterator->getInvalidDeltas()) {
+        $queue->createItem(new ProcessingOperation(ProcessingOperationEnum::PROCESS_CLEANUP, $destination_id, $invalid_deltas));
+      }

Shouldn't the cleanup op come before end?

Fixes #9 and re-rolls against 1.x now the other issue is in

Looks good to me!

thanks!

Automatically closed - issue fixed for 2 weeks with no activity.