Split results storage in keyValue per plugin instead of global

Created on 8 January 2025, 4 months ago

Problem/Motivation

After working on โœจ Improve error handling related to Drupal.org endpoint Active , where we were forcing errors, we realised that the errors are cached, even after these are fixed. We thought about invalidating the keyvalue in that case, but we'd need to invalidate it for all plugins, instead of just the failing one, which would be really unefficient.

Steps to reproduce

Force an error in a plugin, visit the browsing page, then fix it and visit the browsing page again, and the error message still displays.

Proposed resolution

Cache results only per plugin instead of in a global object.

๐Ÿ“Œ Task
Status

Active

Version

2.0

Component

Code

Created by

๐Ÿ‡ช๐Ÿ‡ธSpain fjgarlin

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

  • Issue created by @fjgarlin
  • ๐Ÿ‡ช๐Ÿ‡ธSpain fjgarlin
  • First commit to issue fork.
  • Merge request !666Single source queried โ†’ (Merged) created by narendraR
  • Pipeline finished with Failed
    4 months ago
    Total: 482s
    #390466
  • Pipeline finished with Failed
    4 months ago
    Total: 465s
    #390517
  • Pipeline finished with Success
    4 months ago
    Total: 345s
    #390547
  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia narendraR Jaipur, India
  • ๐Ÿ‡ช๐Ÿ‡ธSpain fjgarlin

    Iโ€™m having a look at the code and leaving some comments in the MR. Either Iโ€™m not understanding the logic too well or I think we are losing too much โ€œcachedโ€ information with this approach. Iโ€™ll leave a comment on the issue to have better track of things.

    Right now (before this issue), we are storing all results for all plugins for a particular query in a keyvalue object, all under the same key, for all plugins.

    With this MR, we are storing under that same keyvalue object, only the results of one plugin, so if we are going back and forth between plugins, we'd need to recalculate things.

    I think it'd make more sense to have the "key" of the object being specific to the query and the source. If the call is successful, then store it, and if it isn't (ie: $results->error is not empty), then don't store it (so it goes through the full external query again when the same data is required).

    Possible approach:
    getProjects says Returns projects that match a particular query, from all enabled sources.
    We start and check the cache there, and I think it's wrong, because maybe one plugin was not working at the time.
    We'd need to loop through the sources (quick) and then get the results, which may or may not be cached. ==> this is where we need to introduce the caching, per plugin, not globally.

  • ๐Ÿ‡ฎ๐Ÿ‡ณIndia narendraR Jaipur, India

    Updated method comments and changes done as suggested.
    I think this is ready for review again.

  • Pipeline finished with Success
    4 months ago
    Total: 432s
    #390696
  • Pipeline finished with Failed
    4 months ago
    Total: 400s
    #390910
  • Pipeline finished with Success
    4 months ago
    Total: 409s
    #390915
  • Pipeline finished with Canceled
    4 months ago
    Total: 342s
    #390992
  • Pipeline finished with Success
    4 months ago
    Total: 344s
    #391001
  • Pipeline finished with Success
    4 months ago
    Total: 377s
    #391023
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Thanks @narendrar!

    This is definitely a step forward to untangling the "everything is a single agglomerated set of tabs" architecture that is the albatross around Project Browser's neck. There is more I'd like to do, but it can happen in a follow-up. Baby steps is what it's all about.

  • Pipeline finished with Success
    4 months ago
    Total: 341s
    #391035
  • First commit to issue fork.
  • Pipeline finished with Skipped
    4 months ago
    #391067
  • Pipeline finished with Skipped
    4 months ago
    #391068
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States phenaproxima Massachusetts

    Whew!!!

  • Automatically closed - issue fixed for 2 weeks with no activity.

  • Pipeline finished with Failed
    2 months ago
    Total: 1378s
    #428173
  • Pipeline finished with Failed
    2 months ago
    Total: 1615s
    #428205
  • Pipeline finished with Success
    2 months ago
    Total: 1354s
    #428218
  • Pipeline finished with Canceled
    2 months ago
    Total: 151s
    #429020
  • Pipeline finished with Failed
    2 months ago
    Total: 2709s
    #429024
  • Pipeline finished with Failed
    2 months ago
    Total: 1375s
    #429149
Production build 0.71.5 2024