Incorrect filter group OR behavior, LEFT JOIN changed to INNER JOIN

Created on 1 September 2012, over 12 years ago
Updated 1 May 2024, 7 months ago

Problem/Motivation

As title says - Incorrect filter group OR behavior, INNER JOIN instead of expected LEFT JOIN

From @SoulReceiver #139 comment

Steps to reproduce

  1. Install Drupal
  2. Create an article with the tag 'test'
  3. Create a view of articles and add 'Tags' as an exposed filter with 'Is one of', twice, using OR operator
  4. Set both filters to 'test'
  5. Confirm there are no results, when you should expect the test article to appear

I note that it works as expected if the exposed filters are set to 'Is all of'.
From @pameeela #121 comment

Proposed resolution

  1. Use INNER JOIN only if the join is not for filter group with OR operator
  2. Add tests to replicate the bug and check the fix.

Remaining tasks

---

User interface changes

---

API changes

---

Data model changes

---

Release notes snippet

---

Original summary D7

I have a view, in code, which uses two groups of ANDed filters, with group behaviour OR. Since updating to views 3.5 (from 3.3+173-dev), the filter OR no longer works correctly - I am getting only the nodes that match the second filter group, not nodes that match either group. There is nothing very special about the filters, which operate on node type, status, nid and taxonomy terms. Aggregation is enabled.

I am pulling values from view arguments and supplying those to the filters in hook_views_query_alter(), but if I disable that and use static filter values in the view the OR still fails as above, so I don't think that is the problem.

The relevant query in 3.3 used a LEFT JOIN, this has changed to INNER JOIN in 3.5. See the second join in each query below.

7.x-3.3+173-dev:

SELECT DISTINCT node.title AS node_title, node.nid AS nid, COUNT(field_data_field_related_article.field_related_article_tid) AS field_data_field_related_article_field_related_article_tid, COUNT(field_data_field_related_article.delta) AS field_data_field_related_article_delta, COUNT(field_data_field_related_article.language) AS field_data_field_related_article_language, COUNT(field_data_field_related_article.bundle) AS field_data_field_related_article_bundle
FROM 
{node} node
INNER JOIN {field_data_field_related_article} field_data_field_related_article ON node.nid = field_data_field_related_article.entity_id AND (field_data_field_related_article.entity_type = 'node' AND field_data_field_related_article.deleted = '0')
LEFT JOIN {field_data_field_related_delivery} field_data_field_related_delivery ON node.nid = field_data_field_related_delivery.entity_id AND (field_data_field_related_delivery.entity_type = 'node' AND field_data_field_related_delivery.deleted = '0')
INNER JOIN {node_access} na ON na.nid = node.nid
WHERE (( (node.status = '1') AND (node.type IN  ('article', 'programme')) AND (node.nid != '68') AND (field_data_field_related_article.field_related_article_tid IN  ('54')) )OR( (node.status = '1') AND (node.type IN  ('article', 'programme')) AND (node.nid != '68') AND (field_data_field_related_delivery.field_related_delivery_value IN  ('distance')) ))AND(( (na.gid = '0') AND (na.realm = 'all') )OR( (na.gid = '0') AND (na.realm = 'domain_site') )OR( (na.gid = '3') AND (na.realm = 'domain_id') ))AND (na.grant_view >= '1') 
GROUP BY node_title, nid
ORDER BY node_title ASC

7.x-3.5:

SELECT DISTINCT node.title AS node_title, node.nid AS nid, COUNT(field_data_field_related_article.field_related_article_tid) AS field_data_field_related_article_field_related_article_tid, COUNT(field_data_field_related_article.delta) AS field_data_field_related_article_delta, COUNT(field_data_field_related_article.language) AS field_data_field_related_article_language, COUNT(field_data_field_related_article.bundle) AS field_data_field_related_article_bundle
FROM 
{node} node
INNER JOIN {field_data_field_related_article} field_data_field_related_article ON node.nid = field_data_field_related_article.entity_id AND (field_data_field_related_article.entity_type = 'node' AND field_data_field_related_article.deleted = '0')
INNER JOIN {field_data_field_related_delivery} field_data_field_related_delivery ON node.nid = field_data_field_related_delivery.entity_id AND (field_data_field_related_delivery.entity_type = 'node' AND field_data_field_related_delivery.deleted = '0')
INNER JOIN {node_access} na ON na.nid = node.nid
WHERE (( (node.status = '1') AND (node.type IN  ('article', 'programme')) AND (node.nid != '68') AND (field_data_field_related_article.field_related_article_tid IN  ('54')) )OR( (node.status = '1') AND (node.type IN  ('article', 'programme')) AND (node.nid != '68') AND (field_data_field_related_delivery.field_related_delivery_value IN  ('distance')) ))AND(( (na.gid = '0') AND (na.realm = 'all') )OR( (na.gid = '0') AND (na.realm = 'domain_site') )OR( (na.gid = '3') AND (na.realm = 'domain_id') ))AND (na.grant_view >= '1') 
GROUP BY node_title, nid
ORDER BY node_title ASC
πŸ› Bug report
Status

Fixed

Version

10.2 ✨

Component
ViewsΒ  β†’

Last updated about 12 hours ago

Created by

πŸ‡³πŸ‡ΏNew Zealand john pitcairn

Live updates comments and jobs are added and updated live.
  • needs backport to 7.x-3.x

    The patch should be considered for backport to the 7.x-3.x branch. Note: This tag should generally remain even after the backport has been written, approved, and committed.

Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • First commit to issue fork.
  • last update over 1 year ago
    Custom Commands Failed
  • Status changed to Needs review over 1 year ago
  • πŸ‡·πŸ‡΄Romania vasike Ramnicu Valcea

    new MR available
    the condition to use INNER for join: Only if there is no group with OR operator.
    LEFT JOIN it's by default.

    However this will not work without the solution for https://www.drupal.org/project/drupal/issues/2559961 πŸ› ManyToOneHelper ignores group configuration for some cases Fixed
    which means that it can be really tested.

    And a scenario to reproduce.
    1. Create 2 taxonomies
    and add some terms
    2. Create 2 taxonomy term reference fields
    And create different content
    3. Create a view with 2 filters for those 2 fields
    4. Play with AND and OR operators for those filters

    Again the patch from https://www.drupal.org/project/drupal/issues/2559961 πŸ› ManyToOneHelper ignores group configuration for some cases Fixed it's needed in terms to get the right results.

    p.s. i have no idea why those 2 issues were separated.

  • last update over 1 year ago
    29,402 pass
  • πŸ‡·πŸ‡΄Romania vasike Ramnicu Valcea

    Tests available on related ticket MR

    please check
    https://www.drupal.org/project/drupal/issues/2559961#comment-15085241 πŸ› ManyToOneHelper ignores group configuration for some cases Fixed

  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    If this ticket isn't going to add tests and be dependent on another. Then it should be postponed.

    But even then I think the tests should be different, as the tests on πŸ› ManyToOneHelper ignores group configuration for some cases Fixed should be testing that issue. If the tests are testing the same thing then these tickets would be dups of each other.

  • Status changed to Closed: duplicate over 1 year ago
  • πŸ‡·πŸ‡΄Romania vasike Ramnicu Valcea

    As the solution and tests are in the MR of the related issue ... i would say this is kind of duplicate.

    closed also the MR ... test can't be added without the solution from related issue.

  • Status changed to Needs review 8 months ago
  • πŸ‡¬πŸ‡§United Kingdom SoulReceiver

    This is still an issue, but apply the code in the merge request worked very nicely. Can this be rereviewed?

  • Status changed to Postponed: needs info 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    As is the MR can't be merged as it's closed and No test coverage

    If this ticket were to continue issue summary would have to be updated to standard template.

    But would see if related issue that this was closed for solves the issue, depending on core version you are on.

  • πŸ‡¬πŸ‡§United Kingdom SoulReceiver

    Currently running Drupal 10.2.4, and when creating a View and adding an OR condition of two taxonomy fields, the resulting JOIN's on these tables remains as an INNER rather than a LEFT:

  • Status changed to Needs review 8 months ago
  • πŸ‡·πŸ‡΄Romania vasike Ramnicu Valcea

    It seems the changes for this issue were not included in the related issue commits.

    So i re-opened the MR and added the tests code from there.
    i think we are back to review.

  • Pipeline finished with Success
    8 months ago
    Total: 4043s
    #141505
  • Status changed to Needs work 8 months ago
  • πŸ‡¬πŸ‡§United Kingdom longwave UK

    Thanks for reviving this and adding some tests.

    However, the tests pass with or without the change, see https://git.drupalcode.org/issue/drupal-1766338/-/jobs/1278007 - so are they testing the correct thing?

  • Pipeline finished with Canceled
    8 months ago
    #144134
  • Pipeline finished with Success
    8 months ago
    Total: 1343s
    #144115
  • Status changed to Needs review 8 months ago
  • πŸ‡·πŸ‡΄Romania vasike Ramnicu Valcea

    @longwave thank you for the "guidance"
    Also @SoulReceiver comments helped.
    i remembered some things about this issue.

    indeed the previous tests added tried for the related issue which actually solved this specific case when we have multiple "Has taxonomy term" filters.
    And this multiple uses will bring "LEFT JOIN" from the second filter .. which means the tests will pass anyway.

    The trick ... having the second taxonomy filter as "field filter". It still uses the "taxonomy_index_tid" filter plugin so, I think we're still in the right place and (+) it adds and extra test for taxonomy fields.
    Also removed some tests that weren't that relevant here.

    https://git.drupalcode.org/issue/drupal-1766338/-/jobs/1304700

    I'm more than aware it's not the perfect/complete solution for the fix ... but at least it could save the day.
    To save the (Views Filter Grouping) Universe ... I think another issue is needed ... starting maybe with some kernel tests about those FIlters GROUPING JOINS.

    p.s. Views and tests needs a lot of efforts ... still

  • Status changed to Needs work 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Can the issue summary be updated using standard template.

    Original issue appears to have been written for D7.

  • Status changed to Needs review 8 months ago
  • πŸ‡·πŸ‡΄Romania vasike Ramnicu Valcea

    Summary updated.

  • Pipeline finished with Success
    8 months ago
    Total: 1150s
    #147064
  • Status changed to RTBC 8 months ago
  • πŸ‡ΊπŸ‡ΈUnited States smustgrave

    Issue summary appears much better

    Ran test-only feature

    There was 1 error:
    1) Drupal\Tests\taxonomy\Functional\Views\TaxonomyIndexTidUiTest::testFilterGrouping
    Behat\Mink\Exception\ResponseTextException: Failed asserting that the page matches the pattern '/ioy19583/ui' 1 time(s), 0 found.
    Failed asserting that 0 is identical to 1.
    /builds/issue/drupal-1766338/core/tests/Drupal/Tests/WebAssert.php:734
    /builds/issue/drupal-1766338/core/modules/taxonomy/tests/src/Functional/Views/TaxonomyIndexTidUiTest.php:547
    /builds/issue/drupal-1766338/vendor/phpunit/phpunit/src/Framework/TestResult.php:729
    ERRORS!
    Tests: 5, Assertions: 242, Errors: 1.
    

    So shows coverage.

    Manually testing following the steps I believe I am seeing the issue.

    s an exposed filter with 'Is one of', twice

    I probably would of just used one filter selecting the multiple options but maybe can understand scenarios were you would do it this way.

    Change to ManyToOneHelper makes sense.

    12 years wow, good job on this one!

  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    I've tried to credit people who helped move this issue on and have been thanked on the issue for their contributions. Obviously with so many comments this can be hard to get right - hopefully I have. Please let me know if I've not.

  • Status changed to Fixed 8 months ago
  • πŸ‡¬πŸ‡§United Kingdom alexpott πŸ‡ͺπŸ‡ΊπŸŒ

    Committed and pushed 0dac7cab32 to 11.x and 1c95773ffe to 10.3.x and c1c3ae32ca to 10.2.x. Thanks!

    • alexpott β†’ committed c1c3ae32 on 10.2.x
      Issue #1766338 by vasike, John Pitcairn, sagesolutions, jenlampton,...
    • alexpott β†’ committed 1c95773f on 10.3.x
      Issue #1766338 by vasike, John Pitcairn, sagesolutions, jenlampton,...
    • alexpott β†’ committed 0dac7cab on 11.x
      Issue #1766338 by vasike, John Pitcairn, sagesolutions, jenlampton,...
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024