Clear feed hashes if the expire setting changes

Created on 8 March 2017, almost 8 years ago
Updated 27 September 2023, about 1 year ago

Problem/Motivation

We have a feed with 2 items one 1 month old and one 6 months old.
"Discard items older than" is set to "3 months and 3 weeks".
This keeps only one item in the db.
I change "Discard items older than" to Never and update the feed items.

Expected outcome: I have 2 items
Outcome: I have only 1 item

This is due to the fact that the feed hasn't changed and "if ($has_new_content)" condition is not met, hence the items are not imported again.

Proposed resolution

The hash should be based on the "Discard items older than" value too.

πŸ“Œ Task
Status

Fixed

Version

1.0

Component

Code

Created by

πŸ‡·πŸ‡΄Romania mfernea

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States dcam

    Postponing this on the resolution of πŸ“Œ Convert the Feed hash to the State API Fixed . If that one makes it in, then this will be easier. Really, all we need to do is delete all of the hashes if the expire setting is updated. The only reason we have the hashes in the first place is for optimization, preventing feeds from being processed if they haven't changed.

    I'm not sure if this is a bug either, but I can definitely agree that it's undesirable behavior and we need to make a change.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    Nice πŸ‘Œ

  • Status changed to Active over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States dcam

    The new state service is in, so this issue is unblocked.

    I think we should add a description to the expire setting's form element that explains if it's changed, then it will cause all feeds to be completely reprocessed, potentially resulting in long wait times for imports on sites with large numbers of feeds.

  • πŸ‡ΊπŸ‡ΈUnited States dcam

    @larowlan I need your opinion on this. While I was creating the new state service I just assumed that eventually I'd add a deleteAllHashes() function in the ItemsImporter service to fix this issue. So I started writing it and quickly realized that I was adding a function for other classes to depend on, but it's not included in the interface. In fact, the service's hash functions should probably be protected except that the deprecated Feed functions say to use those instead. I haven't decided on a way to work around this, but I've come up with several options.

    1. Add the hash functions to ItemsImporterInterface - This would be a BC break. And isn't necessarily a good idea anyway because the hash is a function of our default importer, not something overriding services should have to implement.
    2. Add a HashStorage service that the importer and the settings form can both use. Then put the hash functions in it.
      The deprecated Feed functions would be updated to point at these instead.
    3. Add a more general state storage service that can be reused for the other Feed properties that we've talked about converting to state.
    4. To not overthink it and just let the settings form delete all the hashes.

    I think all of the options have potential problems. I'd like to know what you think and if you have other ideas.

  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10

    2 or 3 work for me, or a simple option is a variant of 1 where we add a new interface that ItemsImporter implements so we retain BC

    eg SomeNewInterfaceThatSupportsDeletingHashes

    class ItemImporter implements SomeNewInterfaceThatSupportsDeletingHashes, ItemImporterInterface

  • Status changed to Needs review over 1 year ago
  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 10.1.x + Environment: PHP 8.1 & MariaDB 10.3.22
    last update over 1 year ago
    158 pass
  • πŸ‡ΊπŸ‡ΈUnited States dcam

    I've been thinking about this for days analyzing the options. I just can't see a way to do this by changing or adding an interface without potentially causing a problem. And once I realized that adding a service meant that we'd be adding a service to wrap our own service to fix its shortcomings I realized how ridiculous it sounded.

    So I wrote a patch that implements option 4. The function to clear the hashes is short and sweet. I don't think it has any dependency issue and it's an OK solution. Please let me know what you think.

  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 10.1.x + Environment: PHP 8.1 & MariaDB 10.3.22
    last update over 1 year ago
    158 pass
  • πŸ‡ΊπŸ‡ΈUnited States dcam

    The inline @var comment in the test referenced the wrong class.

  • Status changed to RTBC about 1 year ago
  • πŸ‡¦πŸ‡ΊAustralia larowlan πŸ‡¦πŸ‡ΊπŸ.au GMT+10
    +++ b/src/Plugin/aggregator/processor/DefaultProcessor.php
    @@ -173,12 +186,27 @@ class DefaultProcessor extends AggregatorPluginSettingsBase implements Processor
    +      $this->keyValue->get($id)->delete(ItemsImporter::AGGREGATOR_HASH_KEY);
    

    was going to suggest we use deleteMultiple but then realised they're segmented by feed ID

    Turned out fairly clean in the end

  • Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MariaDB 10.3.22
    21:41
    21:20
    Queued
  • Status changed to Fixed about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States dcam

    was going to suggest we use deleteMultiple but then realised they're segmented by feed ID

    Right?! I was disappointed by that too.

    • dcam β†’ committed 1b2d193d on 2.x
      Issue #2858825 by dcam, larowlan: Clear feed hashes if the expire...
  • Status changed to Needs review about 1 year ago
  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MySQL 8
    last update about 1 year ago
    158 pass
  • πŸ‡ΊπŸ‡ΈUnited States dcam

    I didn't mean to mark it as fixed until I'd patched 1.x, but #20 didn't apply.

    • dcam β†’ committed 418118c4 on 1.x
      Issue #2858825 by dcam, larowlan: Clear feed hashes if the expire...
  • Status changed to Fixed about 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States dcam

    Thanks for the review, @larowlan!

  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024