Use a queue for node create/update indexing

Created on 27 June 2009, over 15 years ago
Updated 10 September 2024, 3 months ago

Search module should add content to the search index when it's created by default. This avoids search being 'broken' on new sites when cron hasn't been set up (or is running more infrequently than an admin is testing search). I added a checkbox to the search settings form, defaulting to on, for sites which want to disable this - since with a big search index it could make node creation very slow. Node updates are still done on cron - we just want content added to the index in the first place so it shows up at all.

✨ Feature request
Status

Needs review

Version

11.0 🔥

Component
Search  →

Last updated 9 days ago

  • Maintained by
  • 🇺🇸United States @pwolanin
Created by

🇬🇧United Kingdom catch

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • 🇬🇧United Kingdom catch

    Thirteen years later there might be a way to do this in a way that the search index would be immediately indexed on sites with automated cron, while faster for sites running queues on a different schedule to cron too, but it'll need ✨ Add an 'instant' queue runner Needs work first. Postponing on that.

  • First commit to issue fork.
  • Merge request !9206Draft: Resolve #504012 "Index content when" → (Open) created by sukr_s
  • Pipeline finished with Failed
    4 months ago
    Total: 453s
    #253626
  • Status changed to Needs review 4 months ago
  • 🇮🇳India sukr_s

    Changes committed. Tests cases will pass only after ✨ Add an 'instant' queue runner Needs work is accepted / merged to 11.x / and updating the fork.

    Setting to NR for reviewing the code changes.

  • Status changed to Needs work 4 months ago
  • 🇺🇸United States smustgrave

    Not sure I'm seeing the need to create performUpdateIndex, unless just from the queueworker. Any BC concern of this breaking existing code now that this code is in a new function?

    Probably can be postponed though on the issue mentioned in #102.

  • 🇮🇳India sukr_s

    performUpdateIndex indexes the given nids. Alternate is to use the updateIndex function in the queue. However this function does a left join on node to determine all the new nodes. This is very expensive operation on large sites. To avoid this expensive query updateIndex was refactored and performUpdateIndex was created. The queue runner invokes performUpdateIndex as the nid is already known saving an expensive db call.

    I don't see any risk from BC perspective since it accepts the nids from the updateIndex function does the same steps as before.

  • 🇬🇧United Kingdom catch

    Splitting the method into two looks good, we need to make sure the queue worker does minimal work.

    I left a couple of comments on the MR. For me this could be committed independently of ✨ Add an 'instant' queue runner Needs work . It wouldn't result in instant search indexing until we have the other issue, but the two issues don't have any hard dependencies on each other otherwise.

  • Status changed to Needs review 4 months ago
  • 🇮🇳India sukr_s

    Changing the NR for MR responses.

  • 🇮🇳India sukr_s

    Related issue #3197264: Node searches fail if index is not complete, and users do not know why → . If the node indexing is being moved entirely to queues, then this scenario also needs to be handled.

  • 🇬🇧United Kingdom catch

    We won't be able to move all node indexing to queues - at least when search is enabled, we need to index every node in the system, which needs the current search index watermark approach.

  • Pipeline finished with Failed
    3 months ago
    Total: 707s
    #279884
  • 🇺🇸United States smustgrave

    Tried cleaning up issue summary but could use 2nd pair of eyes. Original post mentioned something about a new setting but believe that's not the case anymore

    Current MR is still marked as draft and appears to have a test failure.

  • 🇮🇳India sukr_s

    Functionality is dependent on ✨ Add an 'instant' queue runner Needs work as mentioned in #102 → . Once that's Fixed and tests pass, will remove the draft on MR.

Production build 0.71.5 2024