PrivateTempStore::set() will fail if $key is too long

Created on 29 June 2023, about 1 year ago
Updated 11 November 2023, 8 months ago

Problem/Motivation

When PrivateTempStore::set() writes to database table key_value_expire, the name column must be no more than 128 characters. If the $key param to ::set() is too long, it will exceed this.

This problem is the cause of 🐛 Data too long for column 'name' error when exporting as anonymous user Needs review .

The key written to the database comes from ::createkey(), which prepends the value from ::getOwner(). In the case of anonymous users, this is 44 characters.

Proposed resolution

Update ::createkey() function to check length of the key and if it exceeds the set limit then

  1. hash the key
  2. Trim the original key from last (max length - hashed key length)
  3. Prefix the trimmed key from step 2 to hashed key created in step 1

Thus the key length will be limited and we will also preserve as much initial part of original key for the purpose of diagnosing.

Remaining tasks

Agree on what steps to take. Implement.

User interface changes

None.

API changes

If a throw statement is added, the error generated when the key is too long would change.

If the comprehensive fix is implemented, longer values for $key would be a possible.

Data model changes

None.

🐛 Bug report
Status

Needs work

Version

11.0 🔥

Component
Base 

Last updated less than a minute ago

Created by

🇨🇦Canada Liam Morland Ontario, CA 🇨🇦

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @Liam Morland
  • 🇮🇳India ayush.pandey

    Please add steps to reproduce so i can verify the fix as per the expectations.

  • 🇨🇦Canada Liam Morland Ontario, CA 🇨🇦

    Call PrivateTempStore::set() with a long $key param, such as greater than 128 characters. (It can fail with shorter ones too, but that length is sure to fail.)

  • 🇺🇸United States TolstoyDotCom L.A.

    FWIW, I vote for just throwing an exception. 80 characters seems like more than enough. One downside is that someone might test their code using 100 character keys and logged-in users, then when their code is released it blows up when faced with anonymous users, but not everything can be allowed for.

  • 🇨🇦Canada Liam Morland Ontario, CA 🇨🇦

    I encountered the problem in vbo_export module because that generates keys from several variables, such as machine names for Views. If they are long, they add up to too much.

  • 🇮🇳India sidharth_soman Bangalore

    I've added an exception for when the key length exceeds 84 characters. Please review this patch.

  • Status changed to Needs review 11 months ago
  • last update 11 months ago
    29,881 pass
  • 🇨🇦Canada Liam Morland Ontario, CA 🇨🇦
  • Status changed to Needs work 11 months ago
  • 🇺🇸United States smustgrave

    Will need test coverage.

    Also may need a CR as we are throwing a new exception.

  • 🇺🇸United States TolstoyDotCom L.A.

    After $key = $this->createkey($key); in PrivateTempStore::set, $key will be the full key (like "bfCKE9aB6eKLnKPuyGkXIvL5v3iavdtgw48ilCheSYs:somevalue"). That can be 128 chars. So, I think the number in the patch should be changed. It also might be a good idea to use a

    const

    to hold that number and include a comment with an explanation.

  • 🇮🇳India atul4drupal

    Adding the updated patch:

    1) Addition of constant for string length limit as suggested in #9.
    2) Increased length limit to 120 characters as suggested in #9 (Though this limit may be debated and increased or decreased as per consensus arrived at).
    3) Added PhpUnit test.

    Kindly review and suggest.

  • Status changed to Needs review 10 months ago
  • last update 10 months ago
    30,134 pass
  • 🇮🇳India atul4drupal

    forgot top update the issue status.

  • Status changed to Needs work 10 months ago
  • 🇬🇧United Kingdom catch

    Rather than throwing an exception we ought to be able to hash the key if it exceeds a certain length.

  • 🇺🇸United States TolstoyDotCom L.A.

    So, what you mean is if someone tries to save something using a 1000 character key, hash the key before saving? Then, when retrieving that 1000 character key, hash that to get the real key?

  • Status changed to Needs review 10 months ago
  • last update 10 months ago
    30,137 pass
  • 🇮🇳India atul4drupal

    adding the updated patch:
    1) Included hashing of key.
    2) Updated test case for set and get methods.

    Kindly review and suggest.

  • Status changed to Needs work 10 months ago
  • 🇬🇧United Kingdom catch
    1. +++ b/core/lib/Drupal/Core/TempStore/PrivateTempStore.php
      @@ -102,7 +113,12 @@ public function __construct(KeyValueStoreExpirableInterface $storage, LockBacken
         public function get($key) {
      +    if (strlen($key) > self::MAX_TEMPSTORE_KEY_NAME_LENGTH) {
      +      // Hash the value.
      +      $key = Crypt::hashBase64($key);
      +    }
      

      I think the main question is whether to keep the first few characters of the key for debugging etc., this is probably a less common thing to debug than cache IDs though.

    2. +++ b/core/lib/Drupal/Core/TempStore/PrivateTempStore.php
      @@ -118,6 +134,8 @@ public function get($key) {
          *   Thrown when a lock for the backend storage could not be acquired.
      +   * @throws \InvalidArgumentException
      +   *   Thrown when the provided key's length exceeds the maximum allowed length.
          */
      

      This can be removed.

  • Status changed to Needs review 10 months ago
  • last update 10 months ago
    30,137 pass
  • 🇮🇳India atul4drupal

    #16 🐛 PrivateTempStore::set() will fail if $key is too long Needs review .2 - DONE

    #16 🐛 PrivateTempStore::set() will fail if $key is too long Needs review .1 -
    Here I get the point that we would like to preserve few initial words from the original key... Do not see any harm in doing/implementing that but at the same time I also do not see any value addition in doing that either (kindly help me in case I am missing some thing).

    [just thinking aloud here] As in any case the user adding such long tempstore key will generally keep using the same key that is hashed and will always get the desired values. If we attempt to preserve some part from the original key how will that help or what value addition we intend to achieve here!

  • 🇺🇸United States TolstoyDotCom L.A.

    The value addition is let's say you store "flurb" and "zumbo" (pretend those are longer names). Hashing would make those "kq3w12" and "riqjuwq", keeping the first few letters would be "fluw12" and "zumjuwq". That way you could kinda see what they are in case you need to debug something.

  • 🇬🇧United Kingdom catch

    Yes #18 is correct, it's for debugging convenience.

  • Status changed to Needs work 10 months ago
  • 🇺🇸United States smustgrave

    For #16.1

    While at it could the issue summary be updated slightly with the proposed solution. Currently there are 3

  • Status changed to Needs review 10 months ago
  • last update 10 months ago
    30,148 pass
  • 🇮🇳India atul4drupal

    Thanks for clearing the air, adding updated patch:
    1) Updated logic to preserve original key as much possible with hashed suffix.
    2) Updated tests.

    Kindly review and suggest.

  • Status changed to Needs work 10 months ago
  • 🇺🇸United States smustgrave

    For the IS update.

  • Status changed to Needs review 10 months ago
  • last update 10 months ago
    30,150 pass
  • 🇮🇳India atul4drupal

    Updated the patch to limit the code changes to "createKey" method only.
    Realised this while updating the IS.

    kindly review and suggest.

  • 🇨🇦Canada Liam Morland Ontario, CA 🇨🇦

    Remember that the value returned by ::getOwner() can be as much as 44 characters for anonymous users. The patch in #24 only allows for it to be 11 characters.

  • last update 10 months ago
    30,152 pass
  • 🇮🇳India atul4drupal

    Thanks Liam Morland for pointing out the scenario, I have update the patch to take care of the ID value (retured from getOwner) that is prefixed to key.

    Kindly review and suggest.

  • last update 10 months ago
    30,152 pass
  • 🇮🇳India atul4drupal

    apologies for the noise... kindly ignore patch at #26 🐛 PrivateTempStore::set() will fail if $key is too long Needs review .

    Changes introduced:
    re-provisioned for the loss of characters owing to 'ascii_general_ci' collation, this error got introduced in patch at 26 so kindly ignore that.
    Concern raised in #25 has been addressed.

    Kindly review and suggest.

  • Status changed to Needs work 8 months ago
  • The Needs Review Queue Bot tested this issue.

    While you are making the above changes, we recommend that you convert this patch to a merge request . Merge requests are preferred over patches. Be sure to hide the old patch files as well. (Converting an issue to a merge request without other contributions to the issue will not receive credit.)

Production build 0.69.0 2024