Identifiers longer than 63 characters are truncated, causing Views to break on Postgres

Created on 8 September 2009, about 15 years ago
Updated 19 September 2024, 2 months ago

Problem/Motivation

Views can generate SQL alias names in excess of the intended 60 character limit. This can happen particularly when dealing with relationship joins. A too-long value which is not truncated can fail on databases with identifier limits (e.g. 63 characters for Postgres identifiers). A too-long value which is naively truncated can easily conflict with other similarly-truncated aliases based on the same relationships.

Steps to reproduce

Create a View similar to the one described under Original report below.

  1. Follow the instruction for DDEV https://ddev.readthedocs.io/en/latest/users/quickstart/#drupal
  2. But use Postgresql: ddev config --project-type=drupal --php-version=8.3 --docroot=web --database=postgres:16
  3. Add a "List (text)" field to taxonomy term /admin/structure/taxonomy/manage/tags/overview/fields, give a long field name
  4. Create few terms with the new field
  5. Add a "Entity reference" field to article node /admin/structure/types/manage/article/fields, give a long field name
  6. Create a new node content view, add relationship to taxonmy term field from previous bullet
  7. Add filter criteria for field from bullet number three
  8. Face an error

Proposed resolution

When a generated alias exceeds 60 characters, introduce a short hash of that original alias into the name (within the first 60 characters), and truncate the result to 60 characters.

Remaining tasks

*
* Patch review

User interface changes

N/A

API changes

Added: Drupal\views\Plugin\views\query\Sql::sanitizeAlias()

Data model changes

N/A

Release notes snippet

Views SQL aliases which exceed 60 characters are truncated and uniquified based on a hash of the original name.

Original report

We are running our Drupal on PostgreSQL and we have a view with 2 relations (join tables). The Preview SQL is attached below.

Due to the table names and field names, Views generates a field alias longer than 63 characters, node_node_data_field_parent_node_node_data_field_pbcontent_year_field_pbcontent_year_value, causing PostgreSQL not to throw an error, but truncate the name in the resultset. PostgreSQL will only return node_node_data_field_parent_node_node_data_field_pbcontent_year as the column name in the resultset, which causes Views to break, as there is no data in the expected field node_node_data_field_parent_node_node_data_field_pbcontent_year_field_pbcontent_year_value.

SELECT node_node_data_field_parent_node_node_data_field_pbcontent_year.field_pbcontent_year_value AS node_node_data_field_parent_node_node_data_field_pbcontent_year_field_pbcontent_year_value,
   COUNT(DISTINCT(node.nid)) AS num_records
 FROM node node 
 LEFT JOIN content_type_video node_data_field_video_gallery ON node.vid = node_data_field_video_gallery.vid
 INNER JOIN node node_node_data_field_video_gallery ON node_data_field_video_gallery.field_video_gallery_nid = node_node_data_field_video_gallery.nid
 LEFT JOIN content_type_video_gallery node_node_data_field_video_gallery_node_data_field_parent_node ON node_node_data_field_video_gallery.vid = node_node_data_field_video_gallery_node_data_field_parent_node.vid
 INNER JOIN node node_node_data_field_parent_node ON node_node_data_field_video_gallery_node_data_field_parent_node.field_parent_node_nid = node_node_data_field_parent_node.nid
 LEFT JOIN content_type_pbcontent node_node_data_field_parent_node_node_data_field_pbcontent_year ON node_node_data_field_parent_node.vid = node_node_data_field_parent_node_node_data_field_pbcontent_year.vid
 WHERE (node.status <> 0) AND (node.type in ('video'))
 GROUP BY node_node_data_field_parent_node_node_data_field_pbcontent_year_field_pbcontent_year_value
  ORDER BY node_node_data_field_parent_node_node_data_field_pbcontent_year_field_pbcontent_year_value DESC

Could anyone provide me with a workaround? Renaming the base fields and thus getting an shorter alias for the column name would work, but is not an option at the moment.
I tried to rewrite the SQL with hook_views_query_alter(), replacing all aliases in $query->fields with an generic one, if the alias is longer than 63 chars. The query then runs fine, but Views2 continues to search for the long columnames in the resultset. Any way I can tell Views2 that it shouldn't expect the data in field "node_node_data_field_parent_node_node_data_field_pbcontent_year_field_pbcontent_year_value" but for example in "generic_1"?

๐Ÿ› Bug report
Status

RTBC

Version

11.0 ๐Ÿ”ฅ

Component
Viewsย  โ†’

Last updated about 4 hours ago

Created by

๐Ÿ‡ฉ๐Ÿ‡ชGermany bkonetzny

Live updates comments and jobs are added and updated live.
  • PostgreSQL

    Particularly affects sites running on the PostgreSQL database.

  • VDC

    Related to the Views in Drupal Core initiative.

Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • ๐Ÿ‡ณ๐Ÿ‡ดNorway steinmb
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand xurizaemon ลŒtepoti, Aotearoa ๐Ÿ
  • ๐Ÿ‡ฉ๐Ÿ‡ชGermany pebosi

    Could you add a re-rolled patch for 10.1 and 10.2 ?

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    #165 applies just fine to 10.1. You can check 10.2 and let us know if it doesn't apply.

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    Following up #166, I think my guess there was correct.

    > It's not immediately obvious what field_test_data_i was originally ... Maybe it was "id"?

    EntityReferenceRelationshipTest::setUp() does this:

        // Create reference from entity_test to entity_test_mul.
        $this->createEntityReferenceField('entity_test', 'entity_test', 'field_test_data', 'field_test_data', 'entity_test_mul');
    
        // Create reference from entity_test_mul to entity_test.
        $this->createEntityReferenceField('entity_test_mul', 'entity_test_mul', 'field_data_test', 'field_data_test', 'entity_test');
    

    Which looks like it's making field_test_data a reference to a entity_test_mul entity, and field_data_test a reference to a entity_test entity.

    Both of those entity classes declare:

     *   entity_keys = {
     *     "id" = "id",
     *     "uuid" = "uuid",
     *     "bundle" = "type",
     *     "label" = "name",
     *     "langcode" = "langcode",
     *   },
    

    So id looks like the only property it might be if the name is just a concatenation of field name and property key.

    And finally, my interpretation for the truncated name is consistent with other appearances of that comment for the field_data_test field:

    3 matches for "Test the forward relationship." in buffer: EntityReferenceRelationshipTest.php
           :
        142:      // Test the forward relationship.
           :      $this->assertEquals(1, $row->entity_test_mul_property_data_entity_test__field_test_data_i);
    -------
           :
        222:      // Test the forward relationship.
           :      $this->assertEquals(1, $row->entity_test_entity_test_mul__field_data_test_id);
    -------
           :
        281:      // Test the forward relationship.
           :      // $this->assertEquals(1, $row->entity_test_entity_test_mul__field_data_test_id);
    

    So I think the way forward is:

    use Drupal\views\Plugin\views\query\Sql;
    ...
          // Test the forward relationship.
          $alias = Sql::sanitizeAlias('entity_test_mul_property_data_entity_test__field_test_data_id')
          $this->assertEquals(1, $row->{$alias});
    
  • Status changed to Needs review 10 months ago
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    So let's give that a whirl (against 10.1.x initially).

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu
  • last update 10 months ago
    Custom Commands Failed
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu
    Running PHPStan on *all* files.
     ------ ---------------------------------------------------------------
      Line   core/modules/views/src/Plugin/views/query/Sql.php
     ------ ---------------------------------------------------------------
      525    Variable $alias in isset() always exists and is not nullable.
    

    Which is complaining about the pre-existing (not part of this patch) code in Drupal\views\Plugin\views\query\Sql::queueTable():

        // If no alias is specified, give it the default.
        if (!isset($alias)) {
          $alias = $this->tables[$relationship][$table]['alias'] . $this->tables[$relationship][$table]['count'];
    ...
    

    The "always exists" is certainly accurate, but on first look I'm not convinced by the "is not nullable" part. PHPStan must have a reason for claiming otherwise (and I'd tend to assume it's right and I'm wrong), but the reason is sufficiently non-obvious that I can only question the value of eliminating safeguard code on that basis.

    By my reading, if we had two calls to queueTable($table, $relationship = NULL, JoinPluginBase $join = NULL, $alias = NULL) using the same table and relationship arguments, and if the second of those calls had a NULL alias argument, then that null could reach the isset() call. Joining to a database table twice with different aliases and join conditions is 100% a valid thing to do in a SQL query (I wouldn't even call it uncommon), so I see no reason why in principle this couldn't happen.

    It does seem like an empty string alias shouldn't be considered valid, though (especially as the alias is used as an array key, and null array keys are coerced to empty strings), so I'm changing that !isset() to empty() so that we can catch that case and hopefully prevent PHP Stan from asking us to delete a pre-existing safeguard.

  • last update 10 months ago
    29,722 pass
  • last update 10 months ago
    Patch Failed to Apply
  • last update 10 months ago
    29,722 pass
  • last update 10 months ago
    Patch Failed to Apply
  • last update 10 months ago
    Patch Failed to Apply
  • last update 10 months ago
    Patch Failed to Apply
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    Re-roll of #179 for 10.2.x and 11.x.

  • last update 10 months ago
    Build Successful
  • last update 10 months ago
    Build Successful
  • last update 10 months ago
    25,832 pass, 1,827 fail
  • Status changed to Needs work 10 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    Recommend turning to MR as patches are being phased out.

    But appears to have several test failures.

    Also issue summary should follow standard issue template.

    Thanks!

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    > Recommend turning to MR as patches are being phased out.

    The MR workflow at Drupal continues to not provide immutable patch URLs which is hugely problematic, so I'm sure the old method isn't going anywhere while that flaw remains. So long as the traditional patch files are supported I'll prefer to keep using them.

    > appears to have several test failures.

    Of the tested Drupal versions (10.1, 10.2, 11.x) only 10.1 seems to be currently passing its own tests (sans patches), and against that version this patch passed testing too. AFAICT the vast number of test failures on 10.2 and 11 are entirely unrelated to this patch, and there doesn't seem any point in re-testing until those branches start passing tests on their own.

    > Also issue summary should follow standard issue template.

    Indeed, that would be good.

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu
  • ๐Ÿ‡ธ๐Ÿ‡ฐSlovakia poker10

    The MR workflow at drupal.org continues to not provide persistent immutable patch URLs which is hugely problematic when it comes to referencing patches externally, so I'm sure the old method isn't going anywhere while that flaw remains. So long as the traditional patch files are supported I'll prefer to keep using them.

    Please follow this issue #3412417: Disable DrupalCI testing โ†’ regarding the DrupalCI deprecation.

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    jweowu โ†’ changed the visibility of the branch 571548-identifiers-longer-than to hidden.

  • Pipeline finished with Failed
    10 months ago
    Total: 203s
    #84513
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    Based on that issue, I'm not confident that the 10.2.x and 11.x core branches are going to pass their traditional testing anytime soon, so I've made a MR for testing #180 on those versions.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    Can you update to 11.x as the target. It can be backported on commit but has to land on 11.x first.

    But see 10.2 is green so thatโ€™s good

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    Unless I'm missing something, the MR workflow more or less necessitates separate branches for testing against each version of Drupal as it's no longer a simple patch that's being applied, so I've created 10.2.x and 11.x branches for this.

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu
  • Pipeline finished with Success
    10 months ago
    Total: 2050s
    #84559
  • Pipeline finished with Success
    10 months ago
    Total: 1701s
    #84562
  • Status changed to Needs review 10 months ago
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    Well, there's a MR test failure for "PHPUnit Functional Javascript 1/2" which recurred on a retry, but there's no Javascript involvement here, so I'm setting this back to Needs Review.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    Hiding patches for clarity

    Issue summary appeared to be updated in #183 so removing that tag

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand xurizaemon ลŒtepoti, Aotearoa ๐Ÿ

    @jweowu the JS test failure is performance related, suggest to hit Retry on the job @ https://git.drupalcode.org/issue/drupal-571548/-/jobs/713304 if you have a button there for that (I don't).

    I see the existing limit was already 60 characters on aliases, and I've read through the code replacing strtolower(substr($alias, 0, 60)); with similar code in a new method. To me it looks like existing alias behaviour should be preserved.

    I did however wonder if any non-obvious changes in behaviour might impact sites which reference existing aliases that approach the limit, and which are handled differently after this change. For example, if a template referenced a long field name, that might require an update to the field in the template. Based on reading the code I don't think this is the case, but I want to mention it for consideration as that would be a breaking change?

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    We could also get rid of the is_string() check. Can the alias be anything else than a string?

    Yes, at minimum the alias argument can be NULL. I can't recall offhand whether there are other possibilities, but there's at least that one, and the only thing the function can cope with is a string, so I think is_string() is the sensible guard.

  • Pipeline finished with Success
    10 months ago
    Total: 3679s
    #89946
  • Status changed to Needs work 9 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    For the open thread

  • Pipeline finished with Failed
    9 months ago
    Total: 177s
    #94443
  • Status changed to Needs review 9 months ago
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu
  • Pipeline finished with Success
    9 months ago
    Total: 603s
    #94451
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    I don't know why it auto-marked that as a draft, but I'd hazard a guess that it was complaining about the fixup commit (which I think is dumb, because this then means I need to force-push changes, rather than trivially squashing the fixups once it's reviewed and without imposing force-pushes on anyone). Anyhow -- no actual changes there.

  • Pipeline finished with Success
    9 months ago
    Total: 498s
    #94590
  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    That was just a phrasing tweak for the code comments.

  • Pipeline finished with Success
    9 months ago
    Total: 595s
    #95394
  • Status changed to Needs work 9 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    Ran test-only feature so removing that tag

    There was 1 error:
    1) Drupal\Tests\field\Kernel\EntityReference\Views\EntityReferenceRelationshipTest::testNoDataTableRelationship
    Error: Call to undefined method Drupal\views\Plugin\views\query\Sql::sanitizeAlias()
    /builds/issue/drupal-571548/core/modules/field/tests/src/Kernel/EntityReference/Views/EntityReferenceRelationshipTest.php:144
    /builds/issue/drupal-571548/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
    ERRORS!
    Tests: 5, Assertions: 111, Errors: 1.

    Left 2 small comments on MR

    Issue summary mentions a TODO to add steps to reproduce

    Tagged for a simple change record for the new sanitizeAlias feature.

    Believe this is super close

  • Pipeline finished with Success
    9 months ago
    Total: 633s
    #98242
  • Status changed to Needs review 2 months ago
  • ๐Ÿ‡ซ๐Ÿ‡ฎFinland sokru

    I wrote the steps to reproduce the issue and created draft change record.

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand jweowu

    Excellent, thank you sokru.

  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    smustgrave โ†’ changed the visibility of the branch 571548-views-alias-truncation-10.2.x to hidden.

  • Status changed to RTBC 2 months ago
  • ๐Ÿ‡บ๐Ÿ‡ธUnited States smustgrave

    So #bugsmash is trying a new thing where we are focusing on a single component for a month and this month it's views, with this issue coming up.

    The test-only feature fails as expected

    1) Drupal\Tests\field\Kernel\EntityReference\Views\EntityReferenceRelationshipTest::testNoDataTableRelationship
    Error: Call to undefined method Drupal\views\Plugin\views\query\Sql::sanitizeAlias()
    /builds/issue/drupal-571548/core/modules/field/tests/src/Kernel/EntityReference/Views/EntityReferenceRelationshipTest.php:144
    /builds/issue/drupal-571548/vendor/phpunit/phpunit/src/Framework/TestResult.php:728
    ERRORS!
    

    This would make sense since the method doesn't exist yet so no real way to test that.

    The issue summary appears very complete.

    Reviewing the MR and new function are properly typehinted and comments.

    I updated the CR to be more clear that a new function is being added.

  • ๐Ÿ‡ณ๐Ÿ‡ฟNew Zealand quietone

    Thanks for working on this old issue!

    I read the IS, comments and the MR. The IS is easy to understand. For the comments it is interesting that a commit was made to Views when it was in contrib but more work is needed in core. daffie points out that migrate has the same problem with the length of table names. So, I refreshed my memory by looking at that MR.

    The first thing I notice is that there are no comments on the code changes regarding the new method. And that there are no explicit tests of the new method. The existing functional tests is testing with just one string.

    I left some comments and questions in the MR.

Production build 0.71.5 2024