Add support for JSON field queries in database API

Created on 31 July 2023, over 1 year ago
Updated 1 August 2023, over 1 year ago

Problem/Motivation

This is a child issue of ✨ [META] Add support for JSON field queries in entity queries Active .

MySQL, PostgreSQL, and SQLite have different syntaxes for getting an element within a JSON field. Given a json field named "x" with the following value:

{a: {b: 1}}

In MySQL, to get at the value "1", you would write:

SELECT x->>$.a.b FROM ...

In PostgreSQL, you would write:

SELECT x->a->>b FROM ...

For SQLite 3.38 or higher, both of the above syntaxes work.

For SQLite less than 3.38 (TBD whether we need to support lower versions in Drupal 11 core, but even if we don't, we'll need to in contrib because RHEL 9 is on 3.34 and RHEL 10 probably won't be released before Drupal 11 is):

SELECT json_extract(x, '$.a.b') FROM ...

Therefore, Drupal needs an API that calling code can use to express this in a generic way, and then let database drivers implement the database-specific syntax.

Steps to reproduce

Proposed resolution

  • Change Drupal\Core\Database\Query\SelectInterface::addField() from type hinting the $field parameter to a string to type hinting it as a string or an interface. We need to come up with the name and other details for this new interface that $field could be. At a minimum, this new interface would need to be able to express that we want a field name (e.g., "x") transformed by a json path (e.g., ['a', 'b']). We'll also need to figure out the best syntax for the json path that all of the database drivers could deal with. An array seems like a reasonable candidate given the syntax needed by PostgreSQL, and the MySQL and SQLite drivers could transform the array to a JSON pointer.
  • Make a similar change to the $field parameter of Drupal\Core\Database\Query\ConditionInterface::condition().
  • Any other places where we need to support a JSON operation?

Remaining tasks

Discuss if the above proposed resolution seems right, or if we want to explore alternate approaches.

User interface changes

API changes

Data model changes

Release notes snippet

✨ Feature request
Status

Active

Version

11.0 πŸ”₯

Component
DatabaseΒ  β†’

Last updated 2 days ago

  • Maintained by
  • πŸ‡³πŸ‡±Netherlands @daffie
Created by

πŸ‡ΊπŸ‡ΈUnited States effulgentsia

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @effulgentsia
  • πŸ‡ΊπŸ‡ΈUnited States bradjones1 Digital Nomad Life

    Laravel has a similar database abstraction layer and supports this through their own implementation of -> operator (abstracted out for the various supported db drivers, which intersects with our own) - see https://laravel.com/api/8.x/Illuminate/Database/Query/Builder.html#metho... and https://laravel.com/docs/10.x/queries#json-where-clauses

    Their equivalent to our DB drivers (now in separate modules) appear to be "grammars". They extend a common Grammar class and there is a lot of special-casing e.g. for Postgres that we could likely draw a lot of inspiration from or directly lift. Laravel is MIT licensed so no issues there.

    What we would want to decide on early-on is whether this is a job for ::condition() in the query builder or if we wanted to take a Laravel-like approach and add JSON-specific/aware conditions. I like this idea for the reason that it makes explicit the requirements for and syntax of the parameters. I'm still wrapping my head around the idea that there is depth/dimension to these kinds of queries, as opposed to the atomic single-column/value construct of a traditional SQL WHERE clause. This might also allow us to avoid changing method signatures and doing deprecation dances.

    Elsewhere in the PHP ORM ecosystem, there is an extension library for Doctrine, https://github.com/ScientaNL/DoctrineJsonFunctions, but it doesn't appear to provide a true abstraction layer.

    When it comes to entity queries (which would come after we get these foundational building blocks done) we will need to address how to merge these concepts with our DBAL's chaining feature. E.g., ->condition('json_field_name.$.json.path.to.subelement', 'value', 'operator'). This is kinda like how you can do entity_reference_field.entity.that_entities_field.property. We'd need to hash that out when the time comes, and I think there is also some greenfield development/policy to be forged on whether we want to support some of the novel operators available to JSON fields that don't have a standard SQL equivalent.

    I think we do need to additionally accomplish in this issue:

    Identify operators to be supported when testing a jsonpath's value. The current ConditionInterface::condition() docs state:

       * @param string|null $operator
       *   The operator to use. Supported for all supported databases are at least:
       *   - The comparison operators =, <>, <, <=, >, >=.
       *   - The operators (NOT) BETWEEN, (NOT) IN, (NOT) EXISTS, (NOT) LIKE.
       *   Other operators (e.g. LIKE, BINARY) may or may not work. Defaults to =.
    

    which seems easy enough to accomplish. We might even say that an MVP for initial core support is to not at all support things like JSON_CONTAINS() and similar. Now that the db drivers are abstracted into modules, this kind of thing can be sandboxed in contrib-land and then added to core, for instance.

Production build 0.71.5 2024