Currency switching doesn't work for anonymous users with empty cart

Problem

Whenever an anyonymous (not logged in) user selects a different currency in the currency switcher block, the page reloads but currency remains the same. Same thing works perfectly after a product is placed in the cart or if the user is logged in.

Steps to reproduce:
- Install Drupal, install and enable Commerce, Commerce Price, Commerce Currency Resolver 8.x-1.41
- In Commerce settings add a Store, enable at least 2 currencies
- Add a Commerce product, set price to something
- In this module's settings set Currency mapping to "Cookie (currency block selector)"
- Place currency block somewhere on the page
- Open product page as anonymous user
- Try switching currency by selecting another currency in the block's dropdown
- Page reloads, currency displayed on the product and shown in the block dropdown remains the same.
- Place something in cart. After the page reloads, your previous currency selection is applied, and currency switching suddenly works as it should.

Further info/tips
- The problem doesn't show on my dev site where page and render cache is disabled in settings.local.php so I suspect this is something cache related.
- The problem also doesn't show if the user is logged in.
- Might be something with sessions / Drupal not actually starting a session for anonymous users.

Proposed resolution

I don't have the knowledge to debug / solve this, sorry.
Maybe make force session start for anonymous users when switching currency? (Or whatever happens on 'add to cart' that makes it work afterwards.)