Unable to write to a bucket that has object lock enabled

Created on 25 April 2023, over 1 year ago
Updated 27 April 2023, over 1 year ago

Problem/Motivation

  • Object lock is a setting on an S3 bucket that enables immutable file storage - ie. write once, read many times.
  • When trying to write to a bucket that has object lock enabled the following error is returned:

Content-MD5 OR x-amz-checksum- HTTP header is required for Put Object requests with Object Lock parameters

Steps to reproduce

  • Create an S3 bucket and enable object lock.
  • Configure S3FS module to write files to that bucket
  • Click Validate within the S3FS settings Actions page

Proposed resolution

It looks like aws-sdk-php has been updates to enable the required header.
See https://github.com/aws/aws-sdk-php/issues/2256 for details.
I propose that we add an extra setting value for object lock which causes the 'add_content_md5' => true parameter to be passed through to the aws client.

✨ Feature request
Status

Active

Version

4.0

Component

Code

Created by

πŸ‡¬πŸ‡§United Kingdom andrew robinson

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @andrew robinson
  • πŸ‡ΊπŸ‡ΈUnited States cmlara

    I believe this should already be available through hook_s3fs_upload_params_alter() and hook_s3fs_copy_params_alter() which is available for sites to make changes (though I don't think validation currently uses those hooks the full writes do use them.)

    Since this is a new parameter on the AWS SDK and our existing API provides options I'm classifying this as a feature request and pushing to 4.x.

    Would you mind providing some more context on reasons for why the buckets are using Object Lock? I normally don't think of most of the content that would be on Drupal as requiring such a feature so it would be helpful to understand the reasons to better determine if we should implement this inside s3fs or if we should leave this as a hook-only feature.

  • πŸ‡¬πŸ‡§United Kingdom andrew robinson

    Thank you.

    My use case isn't entirely typical.  It relates to compliance and the storage of data in a secure vault in case it is required in a legal case some years in the future.   

    To achieve this I set up an S3 bucket and turned on Object Lock and set Compliance mode with a retention period of 7 years.  This prevents any file added to the bucket from being deleted or overwritten for 7 years. 
    My Drupal application takes a form submission, renders it as a PDF and then saves the PDF into the bucket using s3fs.

    Based on your advice I've tried out the hook_s3fs_upload_params_alter hook.   Unfortunately they are executed after the validate routine so never get invoked due to validate failing.  However, after temporarily bypassing the validate function I can see the hook does run and setting   $upload_params['AddContentMD5'] = true; does resolve my problem.

    So here's the solution in my local copy:

    1.  I've added a new setting into settings.php:    $config['s3fs.settings']['object_lock'] = true;

    2.  I've added a hook into my module:

    function mymodule_s3fs_upload_params_alter(array &$upload_params) {
      $config = \Drupal::config('s3fs.settings')->get();
      if (isset($config['object_lock'])) {
        $upload_params['AddContentMD5'] = TRUE;
      }
    }
    

    3. I've patched the validate function in S3fsService.php as follows:

    diff --git a/src/S3fsService.php b/src/S3fsService.php
    index 97c22f8..464368a 100644
    --- a/src/S3fsService.php
    +++ b/src/S3fsService.php
    @@ -160,6 +160,11 @@ public function validate(array $config) {
             $putOptions['CacheControl'] = $config['cache_control_header'];
           }
    
    +      // An MD5 header is required for files stored in an object lock enabled bucket.
    +      if (!empty($config['object_lock'])) {
    +        $putOptions['AddContentMD5'] = true;
    +      }
    +
           $s3->putObject($putOptions);
           $object = $s3->getObject(['Bucket' => $config['bucket'], 'Key' => $key]);
           if ($object) {
    
  • πŸ‡ΊπŸ‡ΈUnited States cmlara

    I've split out the lack of calling hooks during validation into πŸ› Bucket validation does not call hooks. Downport .

    That will allow us to keep this focused on if we want to implement this as an internally supported feature or if we depend upon hook implementations to do so.

    Just a note, don't forget about hook_s3fs_copy_params_alter() as that is used for FileSytemInterface::copy() operations.

Production build 0.71.5 2024