Add support for public:// and private:// takeover

Created on 26 November 2021, over 2 years ago
Updated 6 July 2023, 12 months ago

Problem/Motivation

Currently the module main page indicates public:// and private:// takeover are not supported. However I belive users may have been using it for some time even though it is not formally supported.

Utilizing the module on a public:// or private:// file field causes data to be written to the s3fs_file table as s3://takeover-prefix/path. In the past users of s3fs_cors have been able to get around this due to code added in #2915322: Image styles: Error generating image, missing source file. β†’ which caused the s3fs module to lookup the data against the S3 bucket in real-time if a file was not found in the metadata table, this code was removed by s3fs in #3201248: Disallow realtime unkown file lookup to S3 -- Reverts #2915322 β†’ .

This was discovered in #3247639: Files being written to s3fs_file table with s3:// when they are in file_managed with private:// β†’ where it appears s3fs_cors has worked with public:// and private:// paths for some time as long as s3fs had that fallback code was in place in place.

Early data appears this could be as simple as moving one line down in the module may be enough to add this feature in, though I have not verified if there are any other conflicts inside of s3fs_cors as I'm not as in-tune with its inner workings as I am s3fs.

Steps to reproduce

Use s3fs_cors on a field storing files against public:// or private:// and check the s3fs_file table

Proposed resolution

Officially add support public:// and private:// takeover by solving any remaining faults.

Remaining tasks

User interface changes

None anticipated.

API changes

None anticipated

Data model changes

s3fs_cors will save entries to s3fs_file as public:// or private:// based on Drupal path instead of always using s3:// bucket path.

✨ Feature request
Status

Fixed

Version

1.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States cmlara

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • heddn Nicaragua

    We use this patch on a few projects. I don't recall all the history behind why. But what is left before this can land? Do we need to add an update hook to refresh s3fs metdata cache?

  • πŸ‡ΊπŸ‡ΈUnited States cmlara

    I'm not actually using s3fs_cors on any sites at the moment so it would be much better to draw from others experiences on that portion to be sure no errors exist (I would suggest your own deployment is likely sufficient.)

    Adding the hook is an option. Personally though I'm a bit hesitant to suggest doing a full S3fsService::refreshCache() as that can take from minutes to hours depending on the status of the bucket. That would leave the option of directly modifying the s3fs_file database table or using the s3fs_file table as a source of info to populate calls to S3fsService::writeTemporaryMetadata() (along with the other calls needed to setup the temporary metadata table.) During both of those the sites should ideally be in maintenance mode with no new file creation permitted to avoid any new writes being out of sync with the s3fs_file table.

    Technically per the README sites never should have used s3fs_cors for public:// or private:// however we know they were. Ultimately adding the hook I think is a question of how user friendly should this be for sites that didn't read the README, or did but latter accidentally deployed an unsupported deployment and how much inconvenience (if we do a full cache refresh) we cause for sites that may have always followed the README.

    Sadly this only worked because of bugs that existed in the s3fs module ignoring the s3fs_file cache table, otherwise it likely would not have made it into production deployments.

  • πŸ‡¬πŸ‡§United Kingdom catch

    There's already a UI option to refresh the metadata cache, so I think it's fine to leave that to sites to do themselves.

  • Status changed to RTBC 12 months ago
  • heddn Nicaragua

    I keep putting this off in fear this will break some poor unsuspecting site. But I'm going to take the plunge and commit these changes. If someone's metadata cache goes wonky, they can refresh the case via the UI.

    • heddn β†’ committed 9a46abb0 on 8.x-1.x
      Issue #3251424 by cmlara, heddn, catch: Add support for public:// and...
  • Status changed to Fixed 12 months ago
  • Automatically closed - issue fixed for 2 weeks with no activity.

Production build 0.69.0 2024