When empty_front_page is enabled, config import failes on some scenario's

For me, I first install with an install profile (as a starting point for my sites), then do a config import. I've used some items from the config_installer profile to make sure this is possible.
If empty_front_page is enabled in the config (in core.extensions.yml), the config import succeeds, but not all config is correct afterwards.