Users can access files attached to old revisions of fields

One of the most common issues I have heard from people with drupal sites in relation to files is that users can access outdated files they attached to old revisions of content.

When using drupal core alone, if you use private files then users will be restricted from seeing files that are only attached to non-current revisions of content.

So if I have node revisions enabled and I have a node with a field containing file1.txt and then I update that node and remove file1.txt and replace with file2.txt, users can no longer access file1.txt directly because it isn't on the current revision of that file field.

When using file_entity I can either restrict or deny access to private files using the "View private files" permission, however using this permission it is not possible to allow users to acces private files only on current revisions, because it just does this:

<?php
      if (user_access('view private files', $account)) {
        return $rights[$account->uid][$cache_id][$op] = TRUE;
      }
?>

I think it should do what file_file_download() does with file_get_file_references() and check only current revisions.

Ideally there would be the ability to give some users permission to view any files and other users the ability to only view files attached to the current revision of content.