Significant developer experience issues posting the schema

Created on 11 December 2024, 25 days ago

Problem/Motivation

Everything was working fine on pre-live environments, but after deploy Search API was unable to index content and we were getting these errors in the logs:

Drupal\search_api_solr\SearchApiSolrException while indexing item entity:node/2321:en: Solr endpoint https://search-gateway-proxy-us-central1-c.svc.pantheon.io:443/ bad request (code: 400, body: Exception writing document id 5kb61n-stanford_hs_index-entity:node/2321:en to the index; possible analysis error: cannot change field "tm_X3b_en_rendered_item" from index options=DOCS_AND_FREQS_AND_POSITIONS to inconsistent index options=DOCS_AND_FREQS_AND_POSITIONS_AND_OFFSETS...

We've seen this problem on another site last week, so there must still be something amiss with Pantheon's recent changes to schema posting. But my coworker who dealt with this last week is out sick today, so I'm left to figure it out on my own.

First I try to repost the schema:

$ terminus drush my-site.live -- search-api-pantheon:postSchema
 [notice] Command: my-site.live -- drush search-api-pantheon:postSchema [Exit: 0]

Well that's not helpful. Did it work? There's some search-related entries in watchdog, but this is a busy site, so I'm not sure if they're related. Even if I add `-vvv` there's no helpful info.

Apparently it didn't work, because I'm still getting the same error when indexing content.

I see that it's going to try to auto-post the correct schema, but I have the option of manually giving it a file path. But what's the format? relative to the webroot, code root, or absolute? And should I use the generic Search API schema (I think Search API Pantheon used to come bundled with its own)??? I scour the internet but can't find answers to any of these questions.

I find the Pantheon support ticket that my coworker submitted last week, and there is finally an answer:

terminus drush <site>.live -- search-api-pantheon:postSchema <search-api-server> modules/contrib/search_api_solr/solr-conf-templates/8.x/

It still doesn't give any indication of success or failure, but this time it did indeed work; I can index content successfully.

Proposed resolution

1. Fix the platform issues with auto-posting the schema
2. Add feedback to the command about success/failure.
3. Update the documentation to show the path format, and the most common example (modules/contrib/search_api_solr/solr-conf-templates/8.x/)

Remaining tasks

🐛 Bug report
Status

Active

Version

8.2

Component

Code

Created by

🇨🇦Canada dalin Guelph, 🇨🇦, 🌍

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @dalin
  • 🇨🇦Canada dalin Guelph, 🇨🇦, 🌍
  • 🇵🇭Philippines danreb

    Add feedback to the command about success/failure.

    Adding -vvv in the drush search-api-pantheon:postSchema command, should normally show you a successful upload of the configuration like in this screenshot:

    Note also that there is a 5 minutes window time after the new configuration to take effect, this mean, you need to wait for a few minutes and then you can reset the index tracker and re-index the site.

    Update the documentation to show the path format, and the most common example (modules/contrib/search_api_solr/solr-conf-templates/8.x/)

    if you --help the drush command, it will show you the format, e.g.

    drush search-api-pantheon:postSchema --help

    Fix the platform issues with auto-posting the schema

    This is a bit challenging, not all sites are using the same path and some are using their own Custom Schema and upload it somewhere in the /code directory.

  • 🇨🇦Canada dalin Guelph, 🇨🇦, 🌍

    > Adding -vvv in the drush search-api-pantheon:postSchema command, should normally show you a successful upload of the configuration like in this screenshot:

    ahh, okay I missed that. I was just moving too fast in a bit of a panic.

    > Note also that there is a 5 minutes window time after the new configuration to take effect, this mean, you need to wait for a few minutes and then you can reset the index tracker and re-index the site.

    So this might've been the core problem. Is this documented anywhere? Ideally after I run `postSchema` either via Drush or the GUI.

    > if you --help the drush command, it will show you the format, e.g.
    > drush search-api-pantheon:postSchema --help

    It doesn't on the versions that I'm using:

  • 🇵🇭Philippines danreb

    Dalin, actually that's it, in your screenshot, it show you the available arguments for the command.

    The command drush search-api-pantheon:postSchema does have 2 arguments:

    1. [server_id]
    2. [path]

    So you should construct it like this-> drush search-api-pantheon:postSchema $SERVER_ID $PATH_TO_CONFIGURATION

    e.g.

    drush search-api-pantheon:postSchema pantheon_solr8 /code/web/modules/contrib/search_api_solr/jump-start/solr8/config-set/ -vvv

  • 🇨🇦Canada dalin Guelph, 🇨🇦, 🌍

    @danreb,
    But back to my original bug report:

    > I do have the option of manually giving it a file path. But what's the format? relative to the webroot, code root, or absolute?

    Also, it would be helpful to show what path it will attempt to use if I omit the default.

  • 🇵🇭Philippines danreb

    @Dalin,

    Ok, I'll try to explain this, the Search API Pantheon module does not shipped with a Solr Schema configuration, but the module search_api_solr have the jumpstart configuration file for Solr 8 as you can see here https://git.drupalcode.org/project/search_api_solr/-/tree/4.x/jump-start...

    If you post the Solr Schema without specifying a directory, the module will just use what is shipped in search_api_solr jumpstart folder.

    The one that you used https://git.drupalcode.org/project/search_api_solr/-/tree/4.x/solr-conf-... which is in the solr-conf-templates/8.x seems to be an old configurations and it has a properties defined that is already been removed in the newer version.

    solr.install.dir=../../../.. in the solrcore.properties file.

    In Pantheon, all the code is in /code directory and the website root is in /code/web

    So in my experience, if you supply a path to the Search API Pantheon CMS admin UI in pantheon, you need to specify the code root directory

    e.g.

    /code/web/modules/contrib/search_api_solr/jump-start/solr8/config-set/

    or if you uploaded your Custom Solr Schema in /code/config/solr directory, you should specify it

    /code/config/solr/

  • 🇨🇦Canada dalin Guelph, 🇨🇦, 🌍

    @danreb

    That's great info.

    So it sounds like what we need is

    drush search-api-pantheon::postSchema --help to output something like

    Arguments:
     [server_id] Search API Server ID to post schema for.
     [path]         Path to schema files (Leave empty to use default schema).  The default is /code/web/modules/contrib/search_api_solr/jump-start/solr8/config-set/
    

    and drush search-api-pantheon::postSchema to output something like

    Successfully posted schema files from /code/web/modules/contrib/search_api_solr/jump-start/solr8/config-set/
    Please wait 5 minutes for the changes to take affect before attempting to index content.
    

    and /admin/config/search/search-api/server/pantheon_solr8/pantheon-admin/schema to output something like

    Successfully posted schema files from /code/web/modules/contrib/search_api_solr/jump-start/solr8/config-set/
    Please wait 5 minutes for the changes to take affect before attempting to index content.
    
  • 🇵🇭Philippines danreb

    @Dalin,

    The 5 minutes window time for the new Solr Schema configuration to take effect is not necessarily means it is 5 minutes. The configuration update will happen within a maximum of 5 minutes and can be much shorter depending on the last checked that the platform does.

    But yes, waiting for 5 minutes is safest.

Production build 0.71.5 2024