Add permission or configuration per text format

Problem/Motivation

It would be nice to be able to configure whether the "Media Library" button appears per text format or at least set permissions for it. I have users that need to be able to add simple links, but don't have the need/access to upload media. Choosing existing media is not needed for this level of user.