Add support for unique / primary key constraints composed of multiple fields for Upsert querries, and start using it for config and key/value tables

Created on 7 August 2015, about 10 years ago
Updated 19 July 2020, about 5 years ago

Problem/Motivation

In #2542776: Add an Upsert class β†’ , we added an UPSERT query to the database system which only worked on tables with one unique index / primary key, composed of a single field. This limitation was imposed by the minimum SQLite version supported by Drupal at the time, which lacked proper support for UPSERT queries. In the meantime, SQLite added a real UPSERT in 3.24.

Proposed resolution

For Drupal 9.x we increased the minimum version of SQLite to 8.26 in #3107155: Discuss lowering SQLite version requirement from 3.26 to 3.22 in Drupal 9 β†’ , and one of the main motivations was the improved UPSERT capability, per #3107155-15: Discuss lowering SQLite version requirement from 3.26 to 3.22 in Drupal 9 β†’ .

Now that all three core database drivers support UPSERT queries on unique/primary key constraints composed of multiple fields, we should take advantage of it and replace core's usage of "Merge" queries where possible.

The benefit is that a Merge query is potentially composed of 3 separate queries (SELECT, INSERT and UPDATE), while UPSERTs can do the same thing (in most situations) in a single query.

Remaining tasks

Review.

User interface changes

Nope.

API changes

API addition, TBD.

Data model changes

Nope.

Release notes snippet

TBD.

πŸ“Œ Task
Status

Needs work

Version

11.0 πŸ”₯

Component
DatabaseΒ  β†’

Last updated 3 days ago

  • Maintained by
  • πŸ‡³πŸ‡±Netherlands @daffie
Created by

πŸ‡·πŸ‡΄Romania amateescu

Live updates comments and jobs are added and updated live.
  • PostgreSQL

    Particularly affects sites running on the PostgreSQL database.

  • Performance

    It affects performance. It is often combined with the Needs profiling tag.

Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • last update almost 2 years ago
    Patch Failed to Apply
  • last update almost 2 years ago
    Patch Failed to Apply
  • last update almost 2 years ago
    Patch Failed to Apply
  • πŸ‡ΊπŸ‡ΈUnited States JonMcL Brooklyn, NY

    @amateescu: Thank you for your work on this.

    Unfortunately it no longer applies to 9.5.x or 10.x. Any chance you have rerolled this patch recently?

  • πŸ‡ΊπŸ‡ΈUnited States mlncn Minneapolis, MN, USA

    Stripped down, MySQL only, no tests, try at an updated patch. Sorry will try to do more. Though i do think the adding the upsert multivalue capability and using elsewhere in Drupal can and should be separated issues.

  • Pipeline finished with Failed
    5 days ago
    Total: 150s
    #570001
  • πŸ‡·πŸ‡΄Romania amateescu

    Thanks @daffie for the review! Addressed most points in the MR, replying below to the other ones:

    Can we use the upsert also for Drupal\Core\KeyValueStore\DatabaseStorage::setIfNotExists()?

    Nope, because that method relies on the return status of the merge query, and upsert doesn't support that.

    Can we replace all calls to Drupal\Core\Database\Query\Merge? If so, can we deprecate Merge?

    Not sure, we can explore that in a separate issue.

    The parameter $fields, is that only an array with values or can it also be an array with keys and values?

    It's only an array with values.

    Can we do some performance testing to make sure that with this patch Drupal becomes faster.

    I ran time ddev drush si demo_umami -y three times before and after this change:

    Before:
    0,06s user 0,02s system 0% cpu 22,412 total
    0,06s user 0,02s system 0% cpu 22,000 total
    0,06s user 0,02s system 0% cpu 22,642 total

    After:
    0,05s user 0,03s system 0% cpu 21,223 total
    0,05s user 0,03s system 0% cpu 20,941 total
    0,06s user 0,02s system 0% cpu 20,550 total

    So it definitely improves things :)

  • Pipeline finished with Success
    5 days ago
    Total: 1240s
    #570010
Production build 0.71.5 2024