Transform the architecture to use a dedicated streamwrapper

Created on 7 April 2022, over 2 years ago
Updated 2 August 2024, 4 months ago

Problem/Motivation

For now the module creates a (empty) file with the upload.
It would be better to have a stream wrapper for Cloudflare to send the file to and not store the file on the file system.

Remaining tasks

  1. Investigate how to have a stream wrapper for Cloudflare.
  2. Implement the stream wrapper for the Cloudflare Video Item field type.
πŸ“Œ Task
Status

Fixed

Version

3.0

Component

Code

Created by

πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    I plan to explore addressing ✨ Tus protocol for file uploads Active in the context of creating a stream wrapper for this issue.

  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    I've pushed an interim branch that depends on πŸ“Œ Move the media source from cloudflare_stream_hosted_video to cloudflare_stream Needs review . I'll make an MR when that work is merged and discuss my approach at that point.

  • Status changed to Postponed 7 months ago
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York
  • Status changed to Active 6 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

    Looking forward for the work that you did. I already looked impressive what you showed me last time. We finished the dependent task so this can be continued.

  • Status changed to Needs review 6 months ago
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York
  • Pipeline finished with Success
    6 months ago
    Total: 173s
    #177224
  • Status changed to Needs work 6 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

    @fathershawn Impressive work!

    There are some PHPCS and PHPStan issues that needs resolving. Didn't had the time yet to fully test this.

  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    I've had a run at fixing the lint. I'm pretty sure that all the critical methods in the streamwrapper have automated tests.

  • Pipeline finished with Success
    6 months ago
    Total: 142s
    #179448
  • Pipeline finished with Success
    6 months ago
    Total: 142s
    #179502
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    These are valid methods as far as I can see. Shall I remove the @covers tags?

    ------ ------------------------------------------------------------------------------------ 
      Line   tests/src/Kernel/CloudflareStreamWrapperTest.php                                    
     ------ ------------------------------------------------------------------------------------ 
      95     @covers value                                                                       
             \Drupal\cloudflare_stream\StreamWrapper\CloudflareStreamWrapper::stream_stat()      
             references an invalid method.                                                       
      113    @covers value                                                                       
             \Drupal\cloudflare_stream\StreamWrapper\CloudflareStreamWrapper::stream_stat()      
             references an invalid method.                                                       
      133    @covers value                                                                       
             \Drupal\cloudflare_stream\StreamWrapper\CloudflareStreamWrapper::stream_stat()      
             references an invalid method.                                                       
      156    @covers value                                                                       
             \Drupal\cloudflare_stream\StreamWrapper\CloudflareStreamWrapper::stream_close()     
             references an invalid method.                                                       
      156    @covers value                                                                       
             \Drupal\cloudflare_stream\StreamWrapper\CloudflareStreamWrapper::stream_open()      
             references an invalid method.                                                       
      180    @covers value                                                                       
             \Drupal\cloudflare_stream\StreamWrapper\CloudflareStreamWrapper::stream_metadata()  
             references an invalid method.                                                       
     ------ ------------------------------------------------------------------------------------ 
    
  • Status changed to Needs review 6 months ago
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    It turns out the correct syntax from phpstan's perspective is @covers ::stream_stat and not @covers ::stream_stat()

  • Pipeline finished with Success
    6 months ago
    Total: 143s
    #180393
  • Status changed to Needs work 5 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

    Finally had the time to check this out.

    I've added a media type and with trying to upload a video, I get the next error

    TypeError: Drupal\cloudflare_stream\Service\CloudflareStreamApi::getDetails(): Argument #1 ($identifier) must be of type string, null given, called in /var/www/html/web/modules/contrib/cloudflare_stream/src/Plugin/Field/FieldType/CloudflareVideoItem.php on line 165 in Drupal\cloudflare_stream\Service\CloudflareStreamApi->getDetails() (line 195 of modules/contrib/cloudflare_stream/src/Service/CloudflareStreamApi.php).
    

    I don't have an idea yet why that happens but this is blocking the testing.

  • Pipeline finished with Success
    5 months ago
    Total: 142s
    #204810
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    Here's the code block

    if ($file instanceof File) {
            $videoId = $this->tempStore->get($file->getFileUri());
            $details = $this->cloudflareStreamApi->getDetails($videoId);
    ...
    }
    

    So this means that a videoId is not stored in tempStore in CloudflareStreamWrapper::stream_close. I'm not sure why - I'll ping you in Slack to see about debugging together.

  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    Also I see that Zone was removed [##3446508]

    I thought this was the customer subdomain but maybe something else?

      /**
       * {@inheritdoc}
       */
      public function getExternalUrl() {
        $id = $this->getIdFromUri();
        $subdomain = $this->cloudflareStream->getZoneId();
    
        if (empty($subdomain) || empty($id)) {
          return '';
        }
        return "https://$subdomain.cloudflarestream.com/$id";
      }
    

    Was this something else?

  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    More follow-up on #17 - I just reproduced your error on a test upload and #18 makes me wonder what else may have shifted so I'll dig in via xdebug and report back

  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    The field setting in my local site reverted the storage URI to public files. I'm not sure why, but perhaps from testing some other things with this branch no longer present?

    In this branch, I change the default URI scheme for the field to our streamwrapper. I think we should go further and unset any other stream wrappers so that there is only our streamwrapper as a storage option.

    I also need a way to compute the external url for the video. Can we add an explicit customer subdomain input back into the settings?

  • Pipeline finished with Success
    5 months ago
    Total: 141s
    #204939
  • Pipeline finished with Success
    5 months ago
    #204963
  • Pipeline finished with Success
    5 months ago
    #207704
  • Pipeline finished with Success
    5 months ago
    Total: 140s
    #207712
  • Pipeline finished with Success
    5 months ago
    Total: 141s
    #207730
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    I've added a subdomain setting in place of the old zone id

    I also changed the storage settings form to only use the new stream wrapper for storage.
    Creating a new media type in a fresh site results in these storage settings with a proper value for uri_scheme.

    langcode: en
    status: true
    dependencies:
      module:
        - cloudflare_stream
        - file
        - media
    id: media.field_media_cloudflare_stream
    field_name: field_media_cloudflare_stream
    entity_type: media
    type: cloudflarevideo
    settings:
      display_field: 0
      display_default: 0
      uri_scheme: cfstream
      target_type: file
    module: cloudflare_stream
    locked: false
    cardinality: 1
    translatable: true
    indexes: {  }
    persist_with_no_fields: false
    custom_storage: false
    
  • Pipeline finished with Success
    5 months ago
    #207733
  • Status changed to Needs review 4 months ago
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York
  • Status changed to Needs work 4 months ago
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    Investigating an error using the most recent version of 10.3: "A path was passed when a fully qualified domain was expected."

  • Status changed to Needs review 4 months ago
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York

    False alarm - I was walking a colleague through the functional flow on my local demo and I must have rebuilt my demo site at some past time - my cloudflare config was empty.

  • Status changed to Needs work 4 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

    Thanks for the updates. I tested the changes and for me everything works as expected. But indeed had to make sure I did a new install and indeed have the correct uri scheme.

    But found that the new config is not being taken care of. I'll try to do my best to add these.

  • Status changed to Needs review 4 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

    Added the config schema's. Let's see if the pipelines succeed and then we can make a 3.x release finally.
    Thanks you very much Shawn for the amazing work done here.

  • Status changed to RTBC 4 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ

    The issues reported in the pipeline are not from this code changes. Lets commit this!

  • Pipeline finished with Skipped
    4 months ago
    #241993
  • Status changed to Fixed 4 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ
  • Status changed to Fixed 4 months ago
  • πŸ‡§πŸ‡ͺBelgium tim-diels Belgium πŸ‡§πŸ‡ͺ
  • πŸ‡ΊπŸ‡ΈUnited States fathershawn New York
Production build 0.71.5 2024