The use of "varchar_ascii" in (some) fields is problematic

Created on 29 October 2024, 2 months ago

Describe your bug or feature request.

The use of the "varchar_ascii" type in some fields can cause fatal errors when creating queries on these fields that include non-ascii characters.
I don't understand why some fields need to be forced to ascii at the database level... Why is this necessary?

If a bug, provide steps to reproduce it from a clean install.

Extended the default "orders" view at /admin/commerce/orders:

  1. create a relationship to the payment
  2. include the used payment gateway as an exposed filter

Enter some non ASCII characters in this exposed filter, for example "mùollie".
This results in a fatal error. (see below)

Drupal\Core\Database\DatabaseExceptionWrapper: Exception in Bestellingen[commerce_orders]: SQLSTATE[HY000]: General error: 1270 Illegal mix of collations (ascii_general_ci,IMPLICIT), (utf8mb4_0900_ai_ci,COERCIBLE), (utf8mb4_0900_ai_ci,COERCIBLE) for operation 'like': SELECT COUNT(*) AS "expression"
FROM
(SELECT 1 AS "expression"
FROM
"commerce_order" "commerce_order"
LEFT JOIN "commerce_payment" "commerce_payment_commerce_order" ON commerce_order.order_id = commerce_payment_commerce_order.order_id
WHERE ("cart" <> :db_condition_placeholder_0) AND ("commerce_payment_commerce_order"."payment_gateway" LIKE :db_condition_placeholder_1 ESCAPE '\\')) "subquery"; Array
(
[:db_condition_placeholder_0] => 1
[:db_condition_placeholder_1] => %mùollie%
[:views_join_condition_0] => 0
)

🐛 Bug report
Status

Active

Version

2.0

Component

Developer experience

Created by

🇧🇪Belgium weseze

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @weseze
  • 🇮🇱Israel jsacksick

    That is Drupal core defining the COLLATION, not Drupal Commerce.

    // Note we check for the "type" key here. "mysql_type" is VARCHAR:
          elseif (isset($spec['type']) && $spec['type'] == 'varchar_ascii') {
            $sql .= ' COLLATE ascii_general_ci';
          }

    See the Mysql schema implementation from Drupal core: https://git.drupalcode.org/project/drupal/-/blob/11.x/core/modules/mysql...

  • 🇧🇪Belgium weseze

    I disagree.
    The type "varchar_ascii" is explicitly set in some commerce schema field definitions. It could also be defined as "varchar", which would default to the collation used by the db/table and "fix" the issue.

    Or do you mean that Drupal core should be responsible for correctly handling these ascii mismatch errors?
    If so, is there a core issue that you can refer me to?

  • 🇮🇱Israel jsacksick

    Ok sorry you're right, replied too quickly here... We indeed explicitly specify "varchar_ascii" for certain fields... I can't recall why TBH... I've never experienced this issue previously though...

    Also FYI, any new bug report / issue should be created against 8.x-2.x.

  • 🇮🇱Israel jsacksick

    A special varchar_ascii type is also available for limiting machine name field to US ASCII characters.

    Ok so that might be why then... Only fields Commerce defines using the "varchar_ascii" type are for fields holding values that are "identifiers" / "machine names"...

  • 🇮🇱Israel jsacksick

    Ok, gave a closer look also at your error message.
    The "problematic" field here is a standard core Entity reference field.

    Core explicitly defines the "varchar_ascii" type for holding the target_id of the entity type the field is targeting. This is done for config entity types (see the else here in EntityReferenceItem::schema()):

        if ($target_type_info->entityClassImplements(FieldableEntityInterface::class) && $properties->getDataType() === 'integer') {
          $columns = [
            'target_id' => [
              'description' => 'The ID of the target entity.',
              'type' => 'int',
              'unsigned' => TRUE,
            ],
          ];
        }
        else {
          $columns = [
            'target_id' => [
              'description' => 'The ID of the target entity.',
              'type' => 'varchar_ascii',
              // If the target entities act as bundles for another entity type,
              // their IDs should not exceed the maximum length for bundles.
              'length' => $target_type_info->getBundleOf() ? EntityTypeInterface::BUNDLE_MAX_LENGTH : 255,
            ],
          ];
        }
    

    So not a lot we can do here... What we did elsewhere in Commerce just follows this pattern.

  • 🇮🇱Israel jsacksick

    See the related issues from 📌 Throw an understandable exception when there is an attempt to load config entities with disallowed characters Needs work . There is likely one of the issues matching your bug report.

  • 🇮🇱Israel jsacksick

    Marking this as won't fix, as for this particular issue reported, any other entity reference field would trigger the same issue elsewhere in Drupal.

Production build 0.71.5 2024