Search API Schema reverts back to 4.2 after dpeloys

Created on 29 February 2024, 11 months ago

Problem/Motivation

Solr schema reverts back to 4.2 after deploys when it previously was 4.3.1 after a custom posting. Additionally any new multidevs that spin up do not use the same schema that the dev environment is using.

From Pantheon support:

After discussing this with our engineers I think I have a better understanding of what’s going on, although I’m not sure what might have prompted the schema to revert in these particular cases.

We have identified the problem that has been causing this sticky solr schema issue. There is a bug in the search_api_pantheon module which is sometimes breaking the solr configuration and causing the core to fail to load, causing the schema to revert to 4.2.

Our engineers are working on an update to the search_api_pantheon module to fix this bug and stop this behavior, so look out for an update to that module soon. Unfortunately, we don’t know of a way to completely avoid this problem until the bug fix is available. If you encounter this issue again, your best best is to try reposting the schema first. If reposting the schema fails and it’s still stuck on 4.2, then our engineers will need to manually fix the solr configuration for the site.

Steps to reproduce

I can't consistently reproduce this as it happens on some environments, not all. But I've been struggling for the last month with multiple projects on Pantheon ever since the Search API Solr upgrade to 4.3 and the needing to re-post a compatible schema.

Proposed resolution

Seems like Pantheon is working on this, but I wanted to note this issue on drupal.org since it could be affecting other users and you may not even notice.

Also it would be great if the default schema could be set in the pantheon.yml so it doesn't just revert back to a whatever the default in the case of any errors. Otherwise our customizations are just lost and there's no way flag that this happened until feature sets are just not working.

πŸ› Bug report
Status

Active

Version

8.1

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States nsciacca

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @nsciacca
  • πŸ‡ΊπŸ‡ΈUnited States nsciacca
  • πŸ‡¬πŸ‡§United Kingdom Sophie.SK

    Ah, I'm so glad you posted this. One of our clients has had this same issue this week, having followed the steps outlined in a Search API Solr issue previously: https://www.drupal.org/project/search_api_solr/issues/3401710 πŸ’¬ Updating to 4.3 Errors Fixed

    It would be good to know what the cause is and when there may be a fix.

  • πŸ‡ΊπŸ‡ΈUnited States nsciacca
  • πŸ‡ΊπŸ‡ΈUnited States nsciacca

    Update: one of our sites randomly reverted back to 4.2.10 on 3/1 even though there was no deploy. This is a huge problem as it prevents new items from being indexed and there's no notice or warning other than our customers reporting missing content.

  • πŸ‡ΊπŸ‡ΈUnited States nsciacca
  • πŸ‡¬πŸ‡§United Kingdom Sophie.SK

    Since this causes a reduction in service for clients, and can happen randomly, I'm raising the priority to "Major".

    Happy to help out testing any patches.

  • πŸ‡ΊπŸ‡ΈUnited States nsciacca

    I've been communicating with Pantheon's support staff and they've said:

    Good day, for an update, Pantheon Platform Engineers already released an update to the Search API Pantheon module https://www.drupal.org/project/search_api_pantheon β†’ just today. The latest version 8.1.7 contains fixes for the Sticky Solr Schema.

    I'll be installing this and monitoring on site asap.

  • πŸ‡ΊπŸ‡ΈUnited States Ec1ipsis

    Just a quick heads up, we've released two versions (8.1.7 and 8.1.8) which resolve bugs that caused this issue. Let us know if you still see it after updates!

  • πŸ‡΅πŸ‡­Philippines danreb

    Bumping this up, this is still happening as it seems the default Solr Schema template of Pantheon containers is using Solr Schema 4.2 and that's the reason why creating a new multidev environment is installed with Solr Schema 4.2 instead of the Solr Schema provided by the search_api_solr module.

    Not sure if there is a plan to update the default Solr Schema, or at least fix the reposting of Solr Schema.

  • πŸ‡ΊπŸ‡ΈUnited States bsnodgrass

    We can confirm the Solr Schema on Pantheon (all environments) is 4.2.10 with search_api_pantheon 8.1.8, search_api 8.x-1.34, and search_api_solr 4.3.3

    /admin/config/search/search-api/server/pantheon_solr8 reports the following:
    Schema Version 4.2.10
    Schema drupal-4.2.10-solr-8.x-1
    Minimal required schema version 4.3.0
    Preferred schema version 4.3.3
    Configured Solr Version 8.0.0
    Detected Solr Version 8.11.2

    Per #5 our Solr Schema did not revert from 4.3.1 AFAIK it has been 4.2.10 since we configured our search. Our new content is being indexed on creation. We do have synonyms defined however and those are not working as expected.

  • πŸ‡ΊπŸ‡ΈUnited States dorficus

    To echo what @bsnodgrass mentioned in #11, I did quite a bit of research trying to find what, if anything, was causing the schema to not get fully updated.
    I posted most of my findings on https://www.drupal.org/project/search_api_pantheon/issues/3449701 πŸ’¬ Support for Configuring Synonyms Active , which seems to be related, but the TL;DR is that it appears there is a hidden schema somewhere on the Pantheon platform that is not being overridden by the custom schema that is posted. For more details on my findings, please check the linked issue.

    Until we have access to be able to update the actual config, I don't believe this issue will be solved.

  • πŸ‡ΊπŸ‡ΈUnited States bsnodgrass

    per @danreb on https://www.drupal.org/project/search_api_pantheon/issues/3449701#commen... πŸ’¬ Support for Configuring Synonyms Active

    He confirmed "The default Solr schema in Pantheon platform was set to 4.2.10, currently reposting of Solr Schema on Pantheon was broken (Sticky Solr Schema bug)"

    and recommended. "If you want your custom config to take affect, what you need to do right now is to open a ticket and let the CSE or the platform engineers reposted the config for you in the affected environments."

  • πŸ‡ΊπŸ‡ΈUnited States kwiseman

    To follow up on #11 πŸ› Search API Schema reverts back to 4.2 after deploys or randomly Active : We're now experiencing this issue too. The Solr schema version has spontaneously reverted back to 4.2.10 on all of our environments and multidevs except for Live for a site after being set to 4.3.3.

  • πŸ‡΅πŸ‡­Philippines danreb

    @kwiseman have you opened a support ticket already? For newly created multidev, it will use the default 4.2.10 Solr Schema and will need reposting.

  • πŸ‡΅πŸ‡­Philippines danreb

    Here is the latest update for the issue on this thread, from the Pantheon Platform engineers.

    The root cause of this issue of Solr Schema reversion to default (4.2.10) comes from upgrades and migrating of cores. Migrating should preserve the schema (and other config files), but there is an edge case where a new core is created with the old schema (our default).

    Platform engineers are already tracking this internally and working on the prevention.

    If in case this happens to customer sites, go to the Search API Pantheon Admin page and repost the Solr Schema, wait for at least 5 minutes for the configuration to take effect (Pantheon platform is checking every 5 minutes if there is an update in Solr Schema and then will issue a reload), then reset the tracker (run drush sapi-rt) and re-index the site (drush sapi-i).

    Alternatively, if in case you are not able to update the Solr Schema, open a ticket and a member of the Technical Support Team will attend to it.

  • πŸ‡ΊπŸ‡ΈUnited States tjmoyer

    When I attempt to go to the Search API Pantheon Admin page I get this error and cannot access the page at all:
    TypeError: Drupal\search_api_pantheon_admin\Form\PantheonSolrAdminForm::getViewSolrFile(): Argument #2 ($contents) must be of type string, array given, called in /code/web/modules/composer/search_api_pantheon/modules/search_api_pantheon_admin/src/Form/PantheonSolrAdminForm.php on line 77 in Drupal\search_api_pantheon_admin\Form\PantheonSolrAdminForm->getViewSolrFile() (line 103 of /code/web/modules/composer/search_api_pantheon/modules/search_api_pantheon_admin/src/Form/PantheonSolrAdminForm.php).

    There is an issue posted related to this: https://www.drupal.org/project/search_api_pantheon/issues/3467907 πŸ› Error when trying to access Pantheon Search Admin page Active .

  • πŸ‡ΊπŸ‡ΈUnited States tjmoyer

    Work around for re-posting schema: you can use the module's drush command search-api-pantheon:postSchema to post the correct schema, as long as you have the correct schema in your codebase. I use Docksal and have the schema in the Docksal configuration in a non-public directory so was able to post schema successfully.

    Steps to achieve this:

    1. Apply the plain diff from the Logger misconfigured for Drush 12+ πŸ› Logger misconfigured for Drush 12 Active issue. Without this you'll get a drush error if you're running drush 12 or above.
    2. Run terminus drush [site].[env] search-api-pantheon:postSchema [server_id] [path-to-schema-directory]. The server_id default is "pantheon_solr8". This may look like terminus drush example.dev search-api-pantheon:postSchema pantheon_solr8 ../non-public/schema/directory/path/
  • πŸ‡ΊπŸ‡ΈUnited States les lim

    If you're having trouble accessing the Pantheon Search Admin tab to re-post the schema, you can still probably still do so without Drush. Go directly to the Post Schema form page for your server at this path:

    /admin/config/search/search-api/server/SERVER_NAME/pantheon-admin/schema

  • πŸ‡΅πŸ‡­Philippines danreb

    This is now an incident that is being work on by Pantheon engineers https://status.pantheon.io/incidents/rv8bw0v6rbjy hoping to be resolve soon.

  • πŸ‡ΊπŸ‡ΈUnited States pwtyler

    The update just released in 8.2.0 should mitigate the reverts.

    8.2.0 adds automatic core reloads after schema updates and reload functionality through both UI and Drush commands. Installing 8.2 and running drush search-api-pantheon:postSchema [server_id] [path-to-schema-directory] again (or posting from the UI) should maintain the index integrity in the future.

  • πŸ‡΅πŸ‡­Philippines danreb

    Pantheon just released Search API Pantheon module version 8.2.0 that should mitigate the reverts issue and also adds automatic core reloads after schema updates. This is also expected customers to post custom Solr schema without the interventions of CSE.

    Pantheon customers can test the module and the incident has been moved to monitoring.

    I will move this issue queue to "needs review"

  • πŸ‡΅πŸ‡­Philippines danreb
  • heddn Nicaragua

    The issues from https://www.drupal.org/project/search_api_pantheon/issues/3461920 πŸ› Logger misconfigured for Drush 12 Active have cropped up again.

    $ terminus drush "${PANTHEON_SITE}.${PANTHEON_ENV}" -- updatedb --no-cache-clear -y
     [error]  TypeError: Drush\Commands\DrushCommands::logger(): Return value must be of type ?Drush\Log\DrushLoggerManager, Drupal\Core\Logger\LoggerChannel returned in Drush\Commands\DrushCommands->logger() (line 77 of /code/vendor/drush/drush/src/Commands/DrushCommands.php) #0 /code/vendor/drush/drush/src/Runtime/ServiceManager.php(480): Drush\Commands\DrushCommands->logger()
    #1 /code/vendor/drush/drush/src/Boot/DrupalBoot8.php(282): Drush\Runtime\ServiceManager->inflect(Object(League\Container\Container), Object(Drupal\search_api_pantheon\Commands\Reload))
    #2 /code/vendor/drush/drush/src/Boot/DrupalBoot8.php(218): Drush\Boot\DrupalBoot8->addDrupalModuleDrushCommands(Object(Drush\Boot\BootstrapManager))
    
  • Status changed to Fixed 6 days ago
  • πŸ‡΅πŸ‡­Philippines danreb

    Sticky Solr Schema and reversion was fixed in Search API Pantheon version 8.2.1+

    Let's put this on Fixed status.

Production build 0.71.5 2024