str_replace throws php error when source contains null

Created on 21 June 2023, over 1 year ago
Updated 17 July 2023, over 1 year ago

Problem/Motivation

With some of the new hardening of PHP, the str_replace process plugin no longer gracefully passes through the source value if it was a NULL or a FALSE. Passing it a non-string value now throws a php error in D10.

This should probably be made to handle the case of a boolean or null a bit more gracefully as those are sometimes common occurrences in data from other sources.

  field_mobile:
    plugin: str_replace
    source: mobile
    search:
      - 'true'
      - 'false'
      - 'null'
    replace:
      - '1'
      - '0'
      - '0'

The code works fine as long as `mobile` contained strings, but when it did not contain a string, php error.

To do this kind of boolean mapping we now use the convert_boolean process plugin, but there are still legitimate times to do str_replace where there may be a NULL or FALSE that gets passed in.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

πŸ› Bug report
Status

Fixed

Version

6.0

Component

Plugins

Created by

πŸ‡ΊπŸ‡ΈUnited States swirt Florida

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @swirt
  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida
  • πŸ‡ΊπŸ‡ΈUnited States edmund.dunn Olympia, WA

    We had this issue pop up again in a different area. This time it was a field that isn't always populated. It comes in with a value of NULL when it isn't populated. Here is a patch. I added a check to verify whether the incoming value is a string or an array of strings, and if it isn't it will throw a MigrateSkipProcessException. I also added a unit test for it.

  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MySQL 5.7
    last update over 1 year ago
    192 pass
  • πŸ‡ΊπŸ‡ΈUnited States edmund.dunn Olympia, WA

    Slight change in approach; this just returns the none string value so that the next process in the chain can continue on. I misunderstood and thought this was what the MigrateSkipProcessException did. This also iterates through an array and acts aon all strings in the array, returning all non-string items as is.

  • Status changed to Needs work over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    This is looking good. Nice work adding the test.

    We have a
    +use Drupal\migrate\MigrateSkipProcessException;

    that snuck in that does not seem to be used.

    It might be good to add some comments in there to show reason for the checks. In a year it would be hard to infer why those are there.

    +    if (!$this->multiple && !is_string($value)) {
             // This can't be processed by str_replace so pass it on through.
    +      return $value;
    +    }
    
    +      foreach($value as $key => $item) {
    +        if (is_string($item)) {
                 // Replace individual item.
    +          $value[$key] = $function($this->configuration['search'], $this->configuration['replace'], $item);
    +        } else {
                 // Can't replace so, so pass it on through.
    +          $value[$key] = $item;
    +        }
    +      }
    
  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    Using this patch is showing failures on migrations where there is some other process plugin that also expects/requires a string. So the patch is only a partial fix, but is allows a breaking change.

    Looking at this a bit closer. php str_replace() handles TRUE and FALSE just fine. It is just the NULL that it chokes on. So I removed the part about bool in the issue description.

    In testing, old php str_replace() would return a empty string if passed a NULL or a FALSE.
    So to keep this running without breaking existing migrations, we would need this to do the following conversions.

    incoming -> outgoing
    NULL -> empty string ''
    FALSE -> empty string ''
    TRUE -> '1'
    integers -> string equivalent

  • Status changed to Active over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States edmund.dunn Olympia, WA

    Ok, I will rework it to do the above.

  • πŸ‡ΊπŸ‡ΈUnited States edmund.dunn Olympia, WA

    Updated patch to allow for the above situations.

  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 10.1.x + Environment: PHP 8.1 & MySQL 5.7
    last update over 1 year ago
    Patch Failed to Apply
  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & sqlite-3.27
    last update over 1 year ago
    Patch Failed to Apply
  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    This is looking good overall.

    This bit seems a little loose though

    +        $item = $this->makeStringIfPossible($item);
    +        if (is_string($item)) {
    +          //Replace the item.
    +          $value[$key] = $function($this->configuration['search'], $this->configuration['replace'], $item);
    +        } else {
    +          //Cannot replace so pass it through.
    +          $value[$key] = $item;
    +        }
    

    What items are we expecting to pass through that would not be a string? The original version of php str_replace returned a string, so if we are passing through something that is not a string, that is going to lead to regressions.
    I don't think we need all the specific cases for TRUE, NULL , FALSE, ... when simply casting them to a string would do it. Here are some quick tests of the various values in php:

    php > print_r(is_string((string) TRUE));
    1
    php > print_r(is_string((string) FALSE));
    1
    php > print_r(is_string((string) NULL));
    1
    php > print_r(is_string((string) 27));
    1
    php > print_r(is_string((string) 27.556));
    1
    php > print_r(is_string((string) "hello"));
    1
    
    

    All the individual cases become strings like we want them to.

    Also it would be helpful if we follow Drupal patch guidelines β†’ on including the comment number in the patch? It makes it much harder to reference patches when they all have the same file name.

  • πŸ‡ΊπŸ‡ΈUnited States edmund.dunn Olympia, WA

    Good point @swirt! Made the relevant changes as this simplifies things.

  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MySQL 5.7
    last update over 1 year ago
    185 pass, 1 fail
  • Status changed to Needs review over 1 year ago
  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.0 & MySQL 5.7
    last update over 1 year ago
    Patch Failed to Apply
  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    #10 is looking great @edmund.dunn, thank you for tackling this so quickly.

  • Open in Jenkins β†’ Open on Drupal.org β†’
    Core: 9.5.x + Environment: PHP 8.1 & MySQL 8
    last update over 1 year ago
    195 pass
  • πŸ‡ΊπŸ‡ΈUnited States edmund.dunn Olympia, WA

    This should fix the tests.

  • πŸ‡ΊπŸ‡ΈUnited States skyriter

    I have run multiple migrations successfully with this patch in my local development environment.

  • Status changed to RTBC over 1 year ago
  • πŸ‡ΊπŸ‡ΈUnited States skyriter

    The records with a null field that had been failing previously are passing once again during migration.

  • Status changed to Fixed over 1 year ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024