token_or module fails without explicit 'clear' option in token service replacement

6run0 → changed the visibility of the branch 3519448-tokenor-module-fails to hidden.

6run0 → changed the visibility of the branch 3519448-tokenor-module-fails to active.

@6run0 thank you for the bug report and issue fork with a potential fix. In order for it to be considered, please add tests that demonstrate both the bug and the fix.

Hey yall,
I’m getting an issue when using [3519448-tokenor-module-fails] to send Twig-based mail templates via Symfony Mailer: the HTML comments in my templates are being stripped out entirely.

What I’m seeing
Any HTML comment of the form <!--[if mso | IE]>…<![endif]--> is picked up by the token detector and removed from the rendered message. For example, a snippet like:

<tr>
  <td><!--[if mso | IE]><table><tr><td><![endif]-->
    <div>
      <table>
        <tbody>
          <tr>
            ...more tables...
          </tr>
        </tbody>
      </table>
    </div>
  <!--[if mso | IE]></td></tr></table><![endif]--></td>
</tr>

is ending up as:

<tr>
  <td><!--><table><tr><td><![endif]-->
    <div>
      <table>
        <tbody>
          <tr>
            ...more tables...
          </tr>
        </tbody>
      </table>
    </div>
  <!--></td></tr></table><![endif]--></td>
</tr>

Where's the problem
I think the problem lies in the "src/TokenOrTokensPreAlter.php" specifically in "tokensPreAlter" function as shown bellow:

...
preg_match_all("/\[[^\[\]]+\]/", $text, $matches);

if (empty($matches)) {
  return;
}

foreach (reset($matches) as $match) {
  if (strpos($match, '|') !== FALSE) {
    $match_clean = substr(substr($match, 1), 0, -1);
    $sub_tokens = explode('|', $match_clean);
...

1. preg_match_all("/\[[^\[\]]+\]/", $text, $matches); identifies them as tokens
2. strpos($match, '|') !== FALSE identifies them as a token_or

Thanks,
TM

@tiage thanks for the feedback on this issue. Feel free to create a separate issue branch and come up with a solution.

I’ve added tests to demonstrate the problem.
Two functional tests in TokenOrWebformFunctionalBrokenTest were updated to reflect the expected behavior.

@lolgm thanks for the tests. With those passing, I've gone ahead and merged this into dev.

Released in v2.3.2