Setting for mastodon (and others) to show/hide followers/following in UI?

Mastodon seems to have a setting (because I don't think it's in the AP spec) to not show the followers or followees of a user in the UI. Not a bad thing, but let's figure out how it works and expose it as a setting. Question is whether that makes much sense or not, if you directly query the actor info, you will always have access to these collections.

The message that is shown is 'This user has chosen to not make this information available'.