EntityViewDisplay::buildMultiple enforces the langcode even when viewing the default translation

Test to see what the CI says...

This can be replicated with nodes (e.g.: basic pages) by simply adding an untranslated node reference field, adjusting the display to show the rendered entity, and using the following hierarchy and translations of basic pages:
- EN level 0 (translated to FR)
-- EN level 1 (untranslated)
--- EN level 2 (translated to FR)

The only difference with paragraphs is that translations of level 1 are created automatically upon saving the level 0 translations, which we don't get with nodes. But the end result is basically the same.
When viewing FR level 0 with the above setup:
- without the patch: we see EN Level 1, EN Level 2
- with the patch: we see EN Level 1, FR Level 2

I'll try to create a phpunit test when I have some time so it makes more sense.

I've tested this and it works.

I think your approach to the problem is correct.

Agree and tested too, works perfectly !