I think that for some settings in the facet form, we should better check for an instance of SearchApiFacetSourceInterface instead of SearchApiDisplay.
My use case is probably a bit unconventional. I have an external solr server that I interface with solely to retrieve facet filters. The actual data that I display in views is retrieved from an API that is also controlling the solr instance. For various reasons I can't use the SearchAPIDisplay as a facet source for my view, so I had to implement my own facet source class that also extends FacetSourcePluginBase and implements SearchApiFacetSourceInterface. As I also need to support hierarchical facets, I wondered why this is supported only if the facet source is an instance of SearchApiDisplay.
I'll attach a patch to make that change as a starting point for discussion. This is probably not needed for a wider audience, but it seems a bit restrictive to limit certain settings to a specific facet source, given all the flexibility that D8 allows.
Needs review
2.0
Code
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.