Refactor Text Processing to Use mb_encode_numericentity for HTML Encoding Compatibility

Created on 16 July 2024, 4 months ago

Updated 31 July 2024, 4 months ago

We utilize a htmlspecialchars_decode(iconv()) which helps encode our text, however there are certain edge cases where:

$dom->loadHTML(htmlspecialchars_decode(iconv('UTF-8', 'ISO-8859-1', htmlentities($text, ENT_COMPAT, 'UTF-8')), ENT_QUOTES));

could result in data loss if there are encoding mismatches when using functions like iconv.

It looks like for PHP 8.2 the accepted solution for this is leveraging:

$dom->loadHTML(mb_encode_numericentity($text, [0x80, 0x10FFFF, 0, ~0], 'UTF-8'));

Let's switch it to use that instead.

🐛 Bug report

Status

Fixed

Version

1.0

Component

Code

Created by

🇺🇸United States j-barnes

Live updates comments and jobs are added and updated live.

Merge Requests

Issue created by @j-barnes
Merge request !1Issue #3461795: Refactor text processing. → (Merged) created by j-barnes
Status changed to Needs review 4 months ago1:55pm 16 July 2024
Comment 4 months ago →
🇺🇸United States j-barnes
Pipeline finished with Skipped
4 months ago
#226929

j-barnes → committed ceb639a9 on 1.0.x

Issue #3461795: Refactor Text Processing to Use mb_encode_numericentity...

Comment 4 months ago →
🇺🇸United States j-barnes
Status changed to Fixed 4 months ago4:02pm 17 July 2024
Comment 4 months ago →
🇺🇸United States j-barnes
Comment 4 months ago →
System Message
Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.71.5 2024