Expose triggering update of media metadata + thumbnail to end users

Created on 3 July 2018, almost 6 years ago
Updated 26 April 2024, about 2 months ago

Problem/Motivation

It's currently impossible for end-users to manually pull metadata associated with media assets from the source into the Drupal media entity, after the entity is created. Similarly, the Thumbnail (which is not a metadata per se) is treated the same way, it is currently only retrieved from the source when the entity is first created.

Several scenarios have shown this is a big limitation, being a prominent example the situation when the source providing the thumbnail is remote (like an YouTube video), and the thumbnail is changed on the remote source after the entity has been created in Drupal. Currently there is no mechanism for users to update the Thumbnail in this scenario. The same is valid for all other metadata attributes that source plugins could define (name, filesize, filemime, width, height, etc).

This issue was spun-off from #2878119: Whether queued or not, update the media thumbnail and metadata before beginning the entity save database transaction .

Proposed resolution

Expose to the UI the possibility of end users with correct permissions to trigger an immediate update of the asset's metadata + thumbnail

Remaining tasks

  • Usability re-review

User interface changes

- New contextual link "Update metadata" is available in the front-end

- New operation dropbutton "Update metadata" is available in the back-end

- New "Update metadata" action is available in media administrative views, as a bulk operation.

API changes

A new public method \Drupal\media\MediaInterface::updateMetadata() is added to the Media interface.

Data model changes

None

Release notes snippet

Adds the ability for users to manually trigger updating metadata on media entities (e.g. updating a more recent YouTube thumbnail).

Feature request
Status

Needs work

Version

11.0 🔥

Component
Media 

Last updated 2 days ago

Created by

🇪🇸Spain marcoscano Barcelona, Spain

Live updates comments and jobs are added and updated live.
  • Needs usability review

    Used to alert the usability topic maintainer(s) that an issue significantly affects (or has the potential to affect) the usability of Drupal, and their signoff is needed. When adding this tag, make it easy to review the issue. Make sure the issue summary describes the problem and the proposed solution. Screenshots usually help a lot! To get sign-off on issues with the "Needs usability review" tag, post about them in the #ux channel on Drupal Slack, and/or attend a UX meeting to demo the patch and get direct feedback from designers/UX folks/product management on next steps. If an issue represents a significant new feature, UI change, or change to the general "user experience" of Drupal, use Needs product manager review instead. See the scope of responsibilities for product managers.

  • Usability

    Makes Drupal easier to use. Preferred over UX, D7UX, etc.

Sign in to follow issues

Merge Requests

Comments & Activities

Not all content is available!

It's likely this issue predates Contrib.social: some issue and comment data are missing.

  • The Needs Review Queue Bot tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

    Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • Status changed to Needs work over 1 year ago
  • 🇦🇲Armenia gfarishyan

    Patch for 9.5.3 users.

  • 🇫🇷France DuaelFr Montpellier, France

    Patch from #122 doesn't apply for me using composer. I could get it to partly apply using the patch utility but then I had fatal errors.
    Here is a new reroll for Drupal 9.5.3

  • 🇺🇸United States SocialNicheGuru

    Is there an update hook to install this file? core/modules/media/config/install/system.action.media_update_metadata.yml

  • last update 12 months ago
    Custom Commands Failed
  • 🇧🇪Belgium joevagyok

    For now, I just went through the patch in #123 to identify the reason why the patch doesn't apply.
    I found some mix-up in there, namely with Media.php and the media.post_update.php parts. I have taken the gitlab pull request as reference and re-rolled the patch over 9.5.9 version of Drupal core.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    Custom Commands Failed
  • 🇧🇪Belgium joevagyok

    Fixed the spelling error in the #126 patch.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    30,341 pass, 3 fail
  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    Patch Failed to Apply
  • 🇧🇪Belgium joevagyok

    Fixed assertion for "Update metadata" operation in test.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    Patch Failed to Apply
  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    28,529 pass
  • 🇧🇪Belgium joevagyok

    Uploading the patch for D9.5.x as well, since #128 fails to apply to that version.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 12 months ago
    30,344 pass
  • Status changed to Needs review 12 months ago
  • 🇺🇸United States smustgrave

    🐛 Remove source from media mappings Needs work may be related as I found when using metadata and trading out an image on a media deletes the files.

  • Status changed to Needs work 11 months ago
  • The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

    This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • last update 11 months ago
    Patch Failed to Apply
  • 🇳🇿New Zealand jweowu

    Re-roll of #129 against Drupal 10.1.x.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & pgsql-14.1
    last update 9 months ago
    Custom Commands Failed
  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 9 months ago
    Custom Commands Failed
  • 🇳🇿New Zealand jweowu

    Re-roll of #129 against Drupal 10.1.x.

    (#134 accidentally included an empty file at a deleted file path.)

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & pgsql-14.1
    last update 9 months ago
    29,646 pass
  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 9 months ago
    29,646 pass
  • last update 9 months ago
    Patch Failed to Apply
  • Status changed to Needs review 9 months ago
  • 🇳🇿New Zealand jweowu

    #120 is maybe a going concern?

  • Status changed to Needs work 9 months ago
  • The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

    This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

    Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

  • 🇩🇪Germany rkoller Nürnberg, Germany

    I came across this issue and applied the patch in #134 Expose triggering update of media metadata + thumbnail to end users Needs work . In general the functionality looks real good and useful. While testing and playing around I've noticed a few details:

    1. The status message when running Update metadata seems solely inform the user that the process started and ran, but not about its potential outcomes:

    a) You are able to update a media item several times in a row but each time you get the same status message returned: Updated metadata on media item x. Just based on that status message, I would assume that the media item is getting updated with new changes on every run. But if you take a look at the Updated-column on /admin/content/media the timestamp for the media item remains the same and shows an update event in the past. A discrepancy and disconnect between the status message and the Updated-column.

    b) After that I was curious and got inspired by #98 Expose triggering update of media metadata + thumbnail to end users Needs work what would be the status message if you update the metadata for a remote video media type's media item where the URL is not reachable because the server is unavailable/down. Not sure if it is a legitimate way to simulate that scenario, but I've simply switched off the WLAN connection while the site was still running locally in DDEV.
    In a Safari window I got You are not connected to the internet for the URL of the remote video's media item. While when I ran Update metadata I've received the same status message "Updated metadata on media item x" and the media item's thumbnail got removed alongside.

    c) Not sure how likely it is that a local source file is getting deleted and the Update metadata process is run, but I've manually removed the original image of an image media type's media item. Again I've received the status message Updated metadata on media item x while the thumbnail got removed as in b).

    I wonder if instead of informing the user that the process updating the metadata got started, would it be more helpful to provide information about the outcome of that process? That way the user would feel more in control. How about the following rough draft, not perfect at all, just wanted to communicate the gist about each scenario:

    • If everything runs through fine with changes in place on the media item's source the current status message could be used.
    • For scenario a) how about something like "Nothing to update. Media item x remains unchanged."
    • For scenario b) something like "The remote source for media item x is currently unavailable. Refreshing the metadata is not possible, try again later."
    • For scenario c) something like "The source file for media item x is missing, unable to refresh the metadata."
      • 2. About the main question if there is a better alternative for describing the action compared to Update metadata? I am uncertain if there is a one or two worded alternative that explains the process clear and unambiguous as a whole. But from my perspective it should be avoided to use loaded terms that might create assumptions, associations, and expectations on the user's end. I think more neutral terms should be used instead preferably. From my perspective Update as well as Metadata are such loaded terms.

        Update sounds familiar in the context of updating for example Core or contrib modules but in the context of entities it feels sort of unusual - in particular for locally stored media items in contrast to the remote video ones. In addition to that the term sort of overlaps with Edit one of the other options in the operations drop button. Personally my initial association was either to add something new to a media item and or it might entail additional pro-active manual steps.
        Metadata, on the other hand, generates certain expectations. For audio you have for example mp3tags for mp3s, for images and locally hosted videos there is EXIF data and documents like PDFs have extensive metadata too. But if you for example try to create a view based on any of the media items it turns out the metadata is mainly about the thumbnail (in Core out of the box only for remote video and image media types) and the metadata attributes (name, filesize, filemime, width, height, etc) as mentioned in the issue summary. That is totally fine but the term metadata is potentially creating certain assumptions and expectations on the user's end and "might" lead into confusion therefore.

        The proposed alternative Refresh (in #10 Expose triggering update of media metadata + thumbnail to end users Needs work and #60 Expose triggering update of media metadata + thumbnail to end users Needs work ) for the term Update might be the more clear and neutral choice. From my perspective the term Refresh implies a more unsupervised and automated process.

        But in both comments I disagree with the proposed alternative for the term metadata. #10 Expose triggering update of media metadata + thumbnail to end users Needs work suggests Refresh from Youtube but as @marcoscano points out in #36 Expose triggering update of media metadata + thumbnail to end users Needs work not every media item is from Youtube therefore that suggestion is not applicable. On the other hand going with the labels suggested in #60 Expose triggering update of media metadata + thumbnail to end users Needs work I consider those too wordy and it also raises the question what kind information is updated?

    Refresh video information
    Refresh image information
    Refresh local video information
    Refresh document information
    

    But I think @devianintegral indirectly provided another viable alternative in #10 Expose triggering update of media metadata + thumbnail to end users Needs work from my point of view. Why not use the term Source instead of Metadata? The term Source is more neutral but at the same time would imply that you synchronize your media items based on the informations available in the media item's source file. So how about the following changes?

    The drop button in the operations column
    All options in the select field are single worded, I wonder if it would make sense to stay consistent here?

    Edit 
    Delete
    Update metadata -> Refresh

    Bulk action options
    For bulk actions I would provide some more context like the other options do and add "from source":

    Delete media
    Publish media 
    Save media
    Unpublish media
    Update metadata -> Refresh from source

    3. As recommended by @benjifisher in #56 Expose triggering update of media metadata + thumbnail to end users Needs work https://www.drupal.org/docs/8/core/modules/media/creating-and-configurin... was already added as noted in #57 Expose triggering update of media metadata + thumbnail to end users Needs work .
    But I wonder if it would make sense to create a help topic for the Update metadata task alongside. That way the user wouldn't have to leave the admin ui, plus it would be an easy and convenient way to explain the underlaying functionality as well what it entails aka what data is being updated/refreshed for the different media types.

    Those were just my personal observations. I think it would be still good to discuss the issue, in particular point 2 about the microcopy, in a larger group in the weekly usability meeting (as suggested by @benjifisher in #108 Expose triggering update of media metadata + thumbnail to end users Needs work ).

  • 🇺🇸United States japerry KVUO

    We've ran into this bug from a slightly different angle with Acquia DAM. We have implemented our own Metadata Sync controller which will attempt to resave the media entity. However the following code in the media entity ensures that if you change the metadata on the external source, it'll never update Drupal because the actual source asset id hasn't changed.

    // Only save value in the entity if the field is empty or if the
    // source field changed.
    if ($translation->hasField($entity_field_name) && ($translation->get($entity_field_name)->isEmpty() || $translation->hasSourceFieldChanged())) {
    $translation->set($entity_field_name, $media_source->getMetadata($translation, $metadata_attribute_name));
    }

    In the meantime we've copied code from core and manually fixed it in our module. Once this issue is fixed, it'll make that code and our metadata sync controller redundant. So +1 to prioritizing a fix here!

  • 🇩🇪Germany marcoka

    Applied #134 succesfully
    I can "update" single items but the "update metadata" is not avaliable in the batch selection

  • 🇧🇪Belgium kensae

    The patch in #134 failed for Drupal 10.2 on the media.routing.yml file, so I've adapted the patch slightly.

  • 🇫🇷France tostinni

    We had a problema applying #141 following the change of getMetada() into getRawMedata in Add a hook to modify oEmbed resource data Needs work #59

    So here is a patch to apply those changes.

  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 5 months ago
    Patch Failed to Apply
  • Open in Jenkins → Open on Drupal.org →
    Environment: PHP 8.1 & MySQL 5.7
    last update 5 months ago
    Patch Failed to Apply
  • 🇫🇷France DuaelFr Montpellier, France

    Patch from #142 was not applying because it has been created inside the core folder instead of at the root of the project.
    This patch solves that issue and applies on 11.x (and 10.2)

  • Pipeline finished with Failed
    4 months ago
    Total: 192s
    #88766
  • 🇫🇷France DuaelFr Montpellier, France

    I rerolled MR !3037 on 11.x based on the code from #134 and #141 (next patches were missing some code).

    The attached patch applies on Drupal 10.2

  • Pipeline finished with Failed
    4 months ago
    Total: 401s
    #88767
  • Pipeline finished with Failed
    4 months ago
    Total: 500s
    #88776
  • 🇭🇺Hungary szato

    Thank you for this feature. Using it on D10.2.2.

    For us, one issue remains: browser cache. E.g. updating YouTube video media, new thumbnails will be created but with the same hash (file name). If the browser cache is enabled, the new thumbnail will not be refreshed in the browser.

    I'm attaching a patch to fix this issue, using timestamp as a suffix for thumbnail files, and care about removing the old ones on the force update. Moved image_path_flush().

    The patch was created from MR3037 diff.

    FYI: as @klausi noted, the scanDirectory() can become unusably slow if the directory contains many files.

  • 🇧🇪Belgium joevagyok

    @szato, we faced the same issue, however we felt changing the logic around the thumbnail name generation might be an overkill and we refrained from solving front-end cache issue with changing the back-end logic.

    We have varnish configured on our sites, which further complicates the issue.
    So we applied the changedTime of the thumbnail entity to the URI of the image style served, as a get parameter, in an md5 encoded format.
    Browser cache relies on the URI too, like varnish does, so cache-busting by get parameter is an acceptable solution which is well separated from the back-end logic of thumbnail generation.
    However, we found that the changedTime on the thumbnail entity is not update when we trigger this action, so that might be something to address here!

  • 🇭🇺Hungary szato

    @joevagyok, if you use hook_preprocess_image_style(), then you can use filemtime($variables['uri']) (the original image file modification time) for changedTime. I think it's better because the entity update time changes every time it's saved again, not just when the file is changed.

  • 🇺🇸United States Luke.Leber Pennsylvania

    Luke.Leber made their first commit to this issue’s fork.

  • 🇺🇸United States Luke.Leber Pennsylvania

    Rebased against 11.x -- hopefully the test bot doesn't freak out 😅.

    Additionally, the concern brought up in #98 is still valid. Upon manual testing, the following procedure results in data loss:

    1. Create a media entity that implements oembed
    2. Disconnect from the public internet
    3. Update the metadata
    4. Notice that the original thumbnail is gone and is not replaced with a new one
  • Pipeline finished with Failed
    3 months ago
    Total: 196s
    #121191
  • Pipeline finished with Failed
    3 months ago
    #121194
  • Pipeline finished with Failed
    3 months ago
    #121195
  • Pipeline finished with Failed
    3 months ago
    Total: 183s
    #121203
  • Pipeline finished with Failed
    3 months ago
    Total: 171s
    #121204
  • 🇺🇸United States firewaller

    +1

  • 🇺🇸United States agentrickard Georgia (US)

    Related to @japerry's comment in #139, I would note that this approach only handles base field values (string, integer, boolean), and there are cases where it would be important to have source data mapped to, say, a taxonomy term reference field.

    The current code can't do that, but I would propose a follow-up issue to make it possible using an event handler.

    I can work on that over in Mapping metadata to reference field Active as a proof-of-concept. Posting here to see if there is an existing issue or indeed, a desire for that kind of feature.

Production build 0.69.0 2024