Disable showing "Custom Breadcrumbs for Taxonomy" for users without 'administer custom breadcrumbs' permission

"Custom Breadcrumbs for Taxonomy" fieldset can also be seen for users without 'administer custom breadcrumbs' permissions too.
I have a user which can edit taxonomy terms but doesn't have the mentioned permission, but can see the following collapsible fieldset on the term edit form:

Currently I have to unset this with an '#after_build' function:

  if(!user_access('administer custom breadcrumbs')){
    unset($form['custom_breadcrumbs_taxonomy_term']);
  }

But this should be done inside the module itself.