I've enabled the message, message_notify, message_subscribe, and message_subscribe_email modules. I created my message types, the flags for subscribe_* and email_*, and added the hook_node_* logic to create the message entities and call message_subscribe_send_message.
The issue I've been running into was that an email was being sent regardless of the email_* flag. I finally tracked this down to the helper function _message_subscribe_add_default_notifiers that is automatically adding the email notifier.
I then tried to track down where to set the default notifiers and turn off email as a default so the email_* flag would take effect. I saw in hook_menu the 'admin/config/system/message-subscribe' configuration page but for the life of me couldn't find / get to it. That was because the 'administer message subscribe' permission isn't available. That permission is defined in the message_subscribe_ui module and I didn't have that enabled.
So it looks like we simply need to move the 'administer message subscribe' permission definition to the message_subscribe module.
Closed: outdated
1.0
Code
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.