Search must be indexed after recipe is applied

If ECA can listen to RecipeAppliedEvent and run cron, then yes, ECA to the rescue. Otherwise I'm not sure how to fix this without code.

I'm not sure if Search API still supports this, but what might be a reasonable workaround for now is to configure the search index to index new content immediately. That should circumvent any need for us to run cron. Although it's not amazing for performance, I'm of the opinion that "not seeming broken" is going to be more important for more folks in this situation.

Indexing content immediately doesn't help for the scenario where a site already has some content and then the site owner decides to apply the search recipe. What's needed is an initial indexing of all configured and existing content when the search_api gets enabled, i.e. the search recipe gets applied.

For that, search_api provides 2 options: using drush does this by using the batch API or using cron indexes as much as possible within 15 seconds. Remaining content will be indexed with every cron run.

For our context discussed here, we can't use the batch API. But the cron option seems to be what we should leverage. As automated cron is enabled in Drupal CMS, we could set the state key system.cron_last to zero (0) which would result in the current page request to run cron which would index existing content for up to 15 seconds. Let's assume that's long enough for the most sites that will do this.

So, how can we set the state key to 0? ECA can do that when config entity search_api.index.content gets created by the recipe.

Shall we give that a try?

I like it. Let's give that a shot and add some test coverage. This seems like another one where we should have a long, geeky comment at the top of the ECA model to explain why we're clearing this obscure state flag.

It also seems like it might be worth it for ECA (in a future release) to have the ability to listen to RecipeAppliedEvent. :)

I've implemented the ECA model and tested it in various ways. It does work. And I've also added a comment into the model to explain what's going on.

However, there are some caveats:

• It works when the recipe gets applied through project browser, as that has the page request that will then run cron for the initial index.
• It does NOT work, when the recipe gets applied on the command line. It needs one more page request in the browser before the content gets indexed.
• The ECA model doing the job has to be available before applying the search recipe; otherwise it will come too late to do its job. For that reason, I've added the ECA model to the starter recipe rather than the search recipe.

Testing this is pretty challenging, I guess. First, we can't test the search recipe in isolation as it lacks content and the ECA model comes from a different recipe. Second, if we could overcome that first issue, I'm not sure how we should test search as I only see an ajax based search box in the front end which returns search results in a drop down. That may be difficult in the test environment. @phenaproxima if you have any ideas for this, I'd be glad if you could build those tests.

It also seems like it might be worth it for ECA (in a future release) to have the ability to listen to RecipeAppliedEvent. :)

Certainly true, created ✨ Add support for event RecipeAppliedEvent Active to address this asap.

What would also be great, if we could add an action plugin to the search_api module to trigger an index run. That would make things much less complicated.

Search recipe sets the items to be indexed immediately for "Content" index. But for content that was already there of course this doesn't help. Maybe some dashboard message should be added instead of setting the last cron run to 0? This will notify the user that the cron needs to be run in order to have all the items indexed. Or maybe the message can lead to the index page where the user needs to click on "Index" button even.

The ECA works as expected.

I am tempted to mark as RTBC, but I am getting a search_api error in the logs.

RuntimeException while trying to render item entity:node/7:en with view mode search_index for search index Content: Failed to start the session because headers have already been sent by "/var/www/html/vendor/symfony/http-foundation/Response.php" at line 402. in Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start() (line 116 of /var/www/html/vendor/symfony/http-foundation/Session/Storage/NativeSessionStorage.php).

#0 /var/www/html/web/core/lib/Drupal/Core/Session/SessionManager.php(128): Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->start()
#1 /var/www/html/web/core/lib/Drupal/Core/Session/SessionManager.php(93): Drupal\Core\Session\SessionManager->startNow()
#2 /var/www/html/vendor/symfony/http-foundation/Session/Storage/NativeSessionStorage.php(282): Drupal\Core\Session\SessionManager->start()
#3 /var/www/html/vendor/symfony/http-foundation/Session/Session.php(201): Symfony\Component\HttpFoundation\Session\Storage\NativeSessionStorage->getBag()
#4 /var/www/html/vendor/symfony/http-foundation/Session/Session.php(221): Symfony\Component\HttpFoundation\Session\Session->getBag()
#5 /var/www/html/vendor/symfony/http-foundation/Session/Session.php(69): Symfony\Component\HttpFoundation\Session\Session->getAttributeBag()
#6 /var/www/html/web/core/modules/views/src/ViewExecutable.php(763): Symfony\Component\HttpFoundation\Session\Session->get()
#7 /var/www/html/web/core/modules/views/src/Plugin/views/display/DisplayPluginBase.php(2268): Drupal\views\ViewExecutable->getExposedInput()
#8 [internal function]: Drupal\views\Plugin\views\display\DisplayPluginBase->elementPreRender()
#9 /var/www/html/web/core/lib/Drupal/Core/Security/DoTrustedCallbackTrait.php(107): call_user_func_array()
#10 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(825): Drupal\Core\Render\Renderer->doTrustedCallback()
#11 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(387): Drupal\Core\Render\Renderer->doCallback()
#12 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(459): Drupal\Core\Render\Renderer->doRender()
#13 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(203): Drupal\Core\Render\Renderer->doRender()
#14 /var/www/html/web/core/lib/Drupal/Core/Template/TwigExtension.php(484): Drupal\Core\Render\Renderer->render()
#15 /var/www/html/web/sites/default/files/php/twig/677c810bd25eb_block.html.twig_jlgEuxprQ6FxcHIzd-7Yrs0zj/lI_tXB4ZaDeNFyEGpKWXYKsXCqtOLzbKbDCm13u7REg.php(100): Drupal\Core\Template\TwigExtension->escapeFilter()
#16 /var/www/html/vendor/twig/twig/src/Template.php(431): __TwigTemplate_faf585ef680721184a0d202cbeb119b8->block_content()
#17 /var/www/html/web/sites/default/files/php/twig/677c810bd25eb_block.html.twig_jlgEuxprQ6FxcHIzd-7Yrs0zj/lI_tXB4ZaDeNFyEGpKWXYKsXCqtOLzbKbDCm13u7REg.php(79): Twig\Template->yieldBlock()
#18 /var/www/html/vendor/twig/twig/src/Template.php(387): __TwigTemplate_faf585ef680721184a0d202cbeb119b8->doDisplay()
#19 /var/www/html/vendor/twig/twig/src/Template.php(343): Twig\Template->yield()
#20 /var/www/html/vendor/twig/twig/src/Template.php(358): Twig\Template->display()
#21 /var/www/html/vendor/twig/twig/src/TemplateWrapper.php(35): Twig\Template->render()
#22 /var/www/html/web/core/themes/engines/twig/twig.engine(33): Twig\TemplateWrapper->render()
#23 /var/www/html/web/core/lib/Drupal/Core/Theme/ThemeManager.php(348): twig_render_template()
#24 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(446): Drupal\Core\Theme\ThemeManager->render()
#25 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(459): Drupal\Core\Render\Renderer->doRender()
#26 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(203): Drupal\Core\Render\Renderer->doRender()
#27 /var/www/html/web/core/lib/Drupal/Core/Template/TwigExtension.php(484): Drupal\Core\Render\Renderer->render()
#28 /var/www/html/web/sites/default/files/php/twig/677c810bd25eb_layout--onecol.html.twig_Pivc4GMqqu0ofsonWmMYENg09/GYMt6AwW0y_AKNAJ8TnBKXHfHbEbLB2nDJ8bgoUCDsg.php(57): Drupal\Core\Template\TwigExtension->escapeFilter()
#29 /var/www/html/vendor/twig/twig/src/Template.php(387): __TwigTemplate_5cef5491d6a171a2b7694c815ce59500->doDisplay()
#30 /var/www/html/vendor/twig/twig/src/Template.php(343): Twig\Template->yield()
#31 /var/www/html/vendor/twig/twig/src/Template.php(358): Twig\Template->display()
#32 /var/www/html/vendor/twig/twig/src/TemplateWrapper.php(35): Twig\Template->render()
#33 /var/www/html/web/core/themes/engines/twig/twig.engine(33): Twig\TemplateWrapper->render()
#34 /var/www/html/web/core/lib/Drupal/Core/Theme/ThemeManager.php(348): twig_render_template()
#35 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(446): Drupal\Core\Theme\ThemeManager->render()
#36 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(459): Drupal\Core\Render\Renderer->doRender()
#37 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(459): Drupal\Core\Render\Renderer->doRender()
#38 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(203): Drupal\Core\Render\Renderer->doRender()
#39 /var/www/html/web/core/lib/Drupal/Core/Template/TwigExtension.php(484): Drupal\Core\Render\Renderer->render()
#40 /var/www/html/web/sites/default/files/php/twig/677c810bd25eb_node.html.twig_Inzr4pATQnun6K99vBAIcaR_m/sB272BwYcSZ9Poqdmx8jvUNIMtw8xfLIlsYusA9EUdQ.php(133): Drupal\Core\Template\TwigExtension->escapeFilter()
#41 /var/www/html/vendor/twig/twig/src/Template.php(387): __TwigTemplate_c26ee512e43821e6bb8940e67a63c302->doDisplay()
#42 /var/www/html/vendor/twig/twig/src/Template.php(343): Twig\Template->yield()
#43 /var/www/html/vendor/twig/twig/src/Template.php(358): Twig\Template->display()
#44 /var/www/html/vendor/twig/twig/src/TemplateWrapper.php(35): Twig\Template->render()
#45 /var/www/html/web/core/themes/engines/twig/twig.engine(33): Twig\TemplateWrapper->render()
#46 /var/www/html/web/core/lib/Drupal/Core/Theme/ThemeManager.php(348): twig_render_template()
#47 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(446): Drupal\Core\Theme\ThemeManager->render()
#48 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(203): Drupal\Core\Render\Renderer->doRender()
#49 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(120): Drupal\Core\Render\Renderer->render()
#50 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(593): Drupal\Core\Render\Renderer->Drupal\Core\Render\{closure}()
#51 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(119): Drupal\Core\Render\Renderer->executeInRenderContext()
#52 /var/www/html/web/modules/contrib/search_api/src/Plugin/search_api/processor/RenderedItem.php(227): Drupal\Core\Render\Renderer->renderInIsolation()
#53 /var/www/html/web/core/lib/Drupal/Component/Utility/DeprecationHelper.php(40): Drupal\search_api\Plugin\search_api\processor\RenderedItem->Drupal\search_api\Plugin\search_api\processor\{closure}()
#54 /var/www/html/web/modules/contrib/search_api/src/Plugin/search_api/processor/RenderedItem.php(225): Drupal\Component\Utility\DeprecationHelper::backwardsCompatibleCall()
#55 /var/www/html/web/modules/contrib/search_api/src/Item/Item.php(281): Drupal\search_api\Plugin\search_api\processor\RenderedItem->addFieldValues()
#56 /var/www/html/web/modules/contrib/search_api/src/Entity/Index.php(1000): Drupal\search_api\Item\Item->getFields()
#57 /var/www/html/web/modules/contrib/search_api/src/Entity/Index.php(948): Drupal\search_api\Entity\Index->indexSpecificItems()
#58 /var/www/html/web/modules/contrib/search_api/search_api.module(117): Drupal\search_api\Entity\Index->indexItems()
#59 /var/www/html/web/core/lib/Drupal/Core/Cron.php(275): search_api_cron()
#60 /var/www/html/web/core/lib/Drupal/Core/Extension/ModuleHandler.php(307): Drupal\Core\Cron->Drupal\Core\{closure}()
#61 /var/www/html/web/core/lib/Drupal/Core/Cron.php(258): Drupal\Core\Extension\ModuleHandler->invokeAllWith()
#62 /var/www/html/web/core/lib/Drupal/Core/Cron.php(97): Drupal\Core\Cron->invokeCronHandlers()
#63 /var/www/html/web/core/lib/Drupal/Core/ProxyClass/Cron.php(75): Drupal\Core\Cron->run()
#64 /var/www/html/web/modules/contrib/automatic_updates/src/CronUpdateRunner.php(126): Drupal\Core\ProxyClass\Cron->run()
#65 /var/www/html/web/core/modules/automated_cron/src/EventSubscriber/AutomatedCron.php(65): Drupal\automatic_updates\CronUpdateRunner->run()
#66 /var/www/html/vendor/symfony/event-dispatcher/EventDispatcher.php(246): Drupal\automated_cron\EventSubscriber\AutomatedCron->onTerminate()
#67 /var/www/html/vendor/symfony/event-dispatcher/EventDispatcher.php(206): Symfony\Component\EventDispatcher\EventDispatcher::Symfony\Component\EventDispatcher\{closure}()
#68 /var/www/html/vendor/symfony/event-dispatcher/EventDispatcher.php(56): Symfony\Component\EventDispatcher\EventDispatcher->callListeners()
#69 /var/www/html/vendor/symfony/http-kernel/HttpKernel.php(114): Symfony\Component\EventDispatcher\EventDispatcher->dispatch()
#70 /var/www/html/web/core/lib/Drupal/Core/StackMiddleware/StackedHttpKernel.php(63): Symfony\Component\HttpKernel\HttpKernel->terminate()
#71 /var/www/html/web/core/lib/Drupal/Core/DrupalKernel.php(683): Drupal\Core\StackMiddleware\StackedHttpKernel->terminate()
#72 /var/www/html/web/index.php(22): Drupal\Core\DrupalKernel->terminate()
#73 {main}

This is an interesting observation @thejimbirch and it doesn't seem to happen every time. Looks like this is an issue with search_api and the automated _cron in a combination. I'm coming to that conclusion because of these lines in the stack trace:

#55 /var/www/html/web/modules/contrib/search_api/src/Item/Item.php(281): Drupal\search_api\Plugin\search_api\processor\RenderedItem->addFieldValues()
#56 /var/www/html/web/modules/contrib/search_api/src/Entity/Index.php(1000): Drupal\search_api\Item\Item->getFields()
#57 /var/www/html/web/modules/contrib/search_api/src/Entity/Index.php(948): Drupal\search_api\Entity\Index->indexSpecificItems()
#58 /var/www/html/web/modules/contrib/search_api/search_api.module(117): Drupal\search_api\Entity\Index->indexItems()
#59 /var/www/html/web/core/lib/Drupal/Core/Cron.php(275): search_api_cron()
#60 /var/www/html/web/core/lib/Drupal/Core/Extension/ModuleHandler.php(307): Drupal\Core\Cron->Drupal\Core\{closure}()
#61 /var/www/html/web/core/lib/Drupal/Core/Cron.php(258): Drupal\Core\Extension\ModuleHandler->invokeAllWith()
#62 /var/www/html/web/core/lib/Drupal/Core/Cron.php(97): Drupal\Core\Cron->invokeCronHandlers()
#63 /var/www/html/web/core/lib/Drupal/Core/ProxyClass/Cron.php(75): Drupal\Core\Cron->run()
#64 /var/www/html/web/modules/contrib/automatic_updates/src/CronUpdateRunner.php(126): Drupal\Core\ProxyClass\Cron->run()
#65 /var/www/html/web/core/modules/automated_cron/src/EventSubscriber/AutomatedCron.php(65): Drupal\automatic_updates\CronUpdateRunner->run()

The way automatic cron works is it allows to send the result of the page request back to the browser and then runs cron. This is great for the user who doesn't have to wait for the cron before they get their result in the browser.

Now, search_api's cron is then obviously rendering the items that it indexes. I have no idea why, but that can lead to such an error as it starts a new session to receive views arguments. Maybe you have an embedded view on a page that gets indexed? In any event, starting a session is not allowed if the response already started sending data to the browser.

I'd say, this same thing can happen for any automated cron run in this context, regardless of this being triggered after the recipe applied. Sounds like a general issue for the search_api module.

✨ Add an 'instant' queue runner Needs work would allow 'index on save' without a user-facing performance penalty - basically same as automated cron but for anything in a queue. However it would rely on the same end of request logic so doesn't help with the above search_api/automated_cron/views issue. How come a view is getting rendered when search indexing a node?

My testing steps were to install Drupal CMS with various content types. The last time with News and Blog I believe.

Then went to Project browser, clicked on the Recipes browser and applied the Search recipe.

I then verified content was created at /admin/content.

Then visited the logs to discover the errors.

Are the views indexing because of the News And Blog listing pages?

General +1 to the approach here, forcing a fresh cron run means there's no impact on perceived performance due to the end of request stuff.

Is there already an issue open about the views search indexing issue? That seems like a general problem.

Merged into 1.x and cherry-picked to 1.0.x. Thanks everyone! Great bug-busting and collaboration afoot, inspires me every day.

Bug busting = finding a problem and assuming ECA can fix it, and then @jurgenhaas doing just that. I love this workflow!!

However it would rely on the same end of request logic so doesn't help with the above search_api/automated_cron/views issue. How come a view is getting rendered when search indexing a node?

Search API can be configured in multiple ways, for example it can be configured to only index certain fields. However the recommended way, which is also implemented here, is that the rendered html is used for indexing content. This way all kinds of alters and preprocesses are included in the final content, as well as all kinds page building techniques can be used for building the final representation of that content.

I opened 🐛 Views checks the session for exposed filters when it shouldn't Active for the views session issue.