Expose triggering update of media metadata + thumbnail to end users

The Needs Review Queue Bot → tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Patch for 9.5.3 users.

Patch from #122 doesn't apply for me using composer. I could get it to partly apply using the patch utility but then I had fatal errors.
Here is a new reroll for Drupal 9.5.3

Is there an update hook to install this file? core/modules/media/config/install/system.action.media_update_metadata.yml

For now, I just went through the patch in #123 to identify the reason why the patch doesn't apply.
I found some mix-up in there, namely with Media.php and the media.post_update.php parts. I have taken the gitlab pull request as reference and re-rolled the patch over 9.5.9 version of Drupal core.

Fixed the spelling error in the #126 patch.

Fixed assertion for "Update metadata" operation in test.

Uploading the patch for D9.5.x as well, since #128 fails to apply to that version.

🐛 Remove source from media mappings Needs work may be related as I found when using metadata and trading out an image on a media deletes the files.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Re-roll of #129 against Drupal 10.1.x.

Re-roll of #129 against Drupal 10.1.x.

(#134 accidentally included an empty file at a deleted file path.)

#120 is maybe a going concern?

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

I came across this issue and applied the patch in #134 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work . In general the functionality looks real good and useful. While testing and playing around I've noticed a few details:

1. The status message when running Update metadata seems solely inform the user that the process started and ran, but not about its potential outcomes:

a) You are able to update a media item several times in a row but each time you get the same status message returned: Updated metadata on media item x. Just based on that status message, I would assume that the media item is getting updated with new changes on every run. But if you take a look at the Updated-column on /admin/content/media the timestamp for the media item remains the same and shows an update event in the past. A discrepancy and disconnect between the status message and the Updated-column.

b) After that I was curious and got inspired by #98 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work what would be the status message if you update the metadata for a remote video media type's media item where the URL is not reachable because the server is unavailable/down. Not sure if it is a legitimate way to simulate that scenario, but I've simply switched off the WLAN connection while the site was still running locally in DDEV.
In a Safari window I got You are not connected to the internet for the URL of the remote video's media item. While when I ran Update metadata I've received the same status message "Updated metadata on media item x" and the media item's thumbnail got removed alongside.

c) Not sure how likely it is that a local source file is getting deleted and the Update metadata process is run, but I've manually removed the original image of an image media type's media item. Again I've received the status message Updated metadata on media item x while the thumbnail got removed as in b).

I wonder if instead of informing the user that the process updating the metadata got started, would it be more helpful to provide information about the outcome of that process? That way the user would feel more in control. How about the following rough draft, not perfect at all, just wanted to communicate the gist about each scenario:

• If everything runs through fine with changes in place on the media item's source the current status message could be used.
• For scenario a) how about something like "Nothing to update. Media item x remains unchanged."
• For scenario b) something like "The remote source for media item x is currently unavailable. Refreshing the metadata is not possible, try again later."
• For scenario c) something like "The source file for media item x is missing, unable to refresh the metadata."

2. About the main question if there is a better alternative for describing the action compared to Update metadata? I am uncertain if there is a one or two worded alternative that explains the process clear and unambiguous as a whole. But from my perspective it should be avoided to use loaded terms that might create assumptions, associations, and expectations on the user's end. I think more neutral terms should be used instead preferably. From my perspective Update as well as Metadata are such loaded terms.

Update sounds familiar in the context of updating for example Core or contrib modules but in the context of entities it feels sort of unusual - in particular for locally stored media items in contrast to the remote video ones. In addition to that the term sort of overlaps with Edit one of the other options in the operations drop button. Personally my initial association was either to add something new to a media item and or it might entail additional pro-active manual steps.
Metadata, on the other hand, generates certain expectations. For audio you have for example mp3tags for mp3s, for images and locally hosted videos there is EXIF data and documents like PDFs have extensive metadata too. But if you for example try to create a view based on any of the media items it turns out the metadata is mainly about the thumbnail (in Core out of the box only for remote video and image media types) and the metadata attributes (name, filesize, filemime, width, height, etc) as mentioned in the issue summary. That is totally fine but the term metadata is potentially creating certain assumptions and expectations on the user's end and "might" lead into confusion therefore.

The proposed alternative Refresh (in #10 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work and #60 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work ) for the term Update might be the more clear and neutral choice. From my perspective the term Refresh implies a more unsupervised and automated process.

But in both comments I disagree with the proposed alternative for the term metadata. #10 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work suggests Refresh from Youtube but as @marcoscano points out in #36 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work not every media item is from Youtube therefore that suggestion is not applicable. On the other hand going with the labels suggested in #60 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work I consider those too wordy and it also raises the question what kind information is updated?

Refresh video information
Refresh image information
Refresh local video information
Refresh document information

But I think @devianintegral indirectly provided another viable alternative in #10 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work from my point of view. Why not use the term Source instead of Metadata? The term Source is more neutral but at the same time would imply that you synchronize your media items based on the informations available in the media item's source file. So how about the following changes?

The drop button in the operations column
All options in the select field are single worded, I wonder if it would make sense to stay consistent here?

Edit 
Delete
Update metadata -> Refresh

Bulk action options
For bulk actions I would provide some more context like the other options do and add "from source":

Delete media
Publish media 
Save media
Unpublish media
Update metadata -> Refresh from source

3. As recommended by @benjifisher in #56 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work https://www.drupal.org/docs/8/core/modules/media/creating-and-configurin... → was already added as noted in #57 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work .
But I wonder if it would make sense to create a help topic for the Update metadata task alongside. That way the user wouldn't have to leave the admin ui, plus it would be an easy and convenient way to explain the underlaying functionality as well what it entails aka what data is being updated/refreshed for the different media types.

Those were just my personal observations. I think it would be still good to discuss the issue, in particular point 2 about the microcopy, in a larger group in the weekly usability meeting (as suggested by @benjifisher in #108 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work ).

We've ran into this bug from a slightly different angle with Acquia DAM. We have implemented our own Metadata Sync controller which will attempt to resave the media entity. However the following code in the media entity ensures that if you change the metadata on the external source, it'll never update Drupal because the actual source asset id hasn't changed.

In the meantime we've copied code from core and manually fixed it in our module. Once this issue is fixed, it'll make that code and our metadata sync controller redundant. So +1 to prioritizing a fix here!

Applied #134 succesfully
I can "update" single items but the "update metadata" is not avaliable in the batch selection

The patch in #134 failed for Drupal 10.2 on the media.routing.yml file, so I've adapted the patch slightly.

We had a problema applying #141 following the change of getMetada() into getRawMedata in ✨ Add a hook to modify oEmbed resource data Needs work #59

So here is a patch to apply those changes.

Patch from #142 was not applying because it has been created inside the core folder instead of at the root of the project.
This patch solves that issue and applies on 11.x (and 10.2)

I rerolled MR !3037 on 11.x based on the code from #134 and #141 (next patches were missing some code).

The attached patch applies on Drupal 10.2

Thank you for this feature. Using it on D10.2.2.

For us, one issue remains: browser cache. E.g. updating YouTube video media, new thumbnails will be created but with the same hash (file name). If the browser cache is enabled, the new thumbnail will not be refreshed in the browser.

I'm attaching a patch to fix this issue, using timestamp as a suffix for thumbnail files, and care about removing the old ones on the force update. Moved image_path_flush().

The patch was created from MR3037 diff.

FYI: as @klausi noted, the scanDirectory() can become unusably slow if the directory contains many files.

@szato, we faced the same issue, however we felt changing the logic around the thumbnail name generation might be an overkill and we refrained from solving front-end cache issue with changing the back-end logic.

We have varnish configured on our sites, which further complicates the issue.
So we applied the changedTime of the thumbnail entity to the URI of the image style served, as a get parameter, in an md5 encoded format.
Browser cache relies on the URI too, like varnish does, so cache-busting by get parameter is an acceptable solution which is well separated from the back-end logic of thumbnail generation.
However, we found that the changedTime on the thumbnail entity is not update when we trigger this action, so that might be something to address here!

@joevagyok, if you use hook_preprocess_image_style(), then you can use filemtime($variables['uri']) (the original image file modification time) for changedTime. I think it's better because the entity update time changes every time it's saved again, not just when the file is changed.

Luke.Leber → made their first commit to this issue’s fork.

Rebased against 11.x -- hopefully the test bot doesn't freak out 😅.

Additionally, the concern brought up in #98 is still valid. Upon manual testing, the following procedure results in data loss:

1. Create a media entity that implements oembed
2. Disconnect from the public internet
3. Update the metadata
4. Notice that the original thumbnail is gone and is not replaced with a new one

+1

Related to @japerry's comment in #139, I would note that this approach only handles base field values (string, integer, boolean), and there are cases where it would be important to have source data mapped to, say, a taxonomy term reference field.

The current code can't do that, but I would propose a follow-up issue to make it possible using an event handler.

I can work on that over in ✨ Mapping metadata to reference field Active as a proof-of-concept. Posting here to see if there is an existing issue or indeed, a desire for that kind of feature.

Note: the latest MR 3037 does not have the browser cache busting that was added in #145 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work .

Rebased on 11.x, I added most of #145 except for removing image_path_flush() in Media, that will affect every type not just OEmbed.

@KarlShea
Regarding your comment "I added most of #145 except for removing image_path_flush() in Media, that will affect every type not just OEmbed."
- That seems like an edge case. You would need to have another media type that is using the thumbnail file of an oEmbed resource. The normal usage would be a 1:1 of media to thumbnail file.

Regarding your comment "Edit: actually I removed #145, it looks like there was some disagreement."
There still needs to be a solution for busting cache - browser, edge cache, reverse proxy.

Options so far for cache busting:
1. The thumbnail generation approach as used in patch #145
Pros:
- Busts through all layers of caching - browser, edge cache reverse proxy.
- Automatically deletes the old file and any image styles for the old file.
- ???
Cons:
- A call to "getMetadata" for the "thumbnail_uri" assumes that the thumbnail is being set on the media entity. If it does not update the media, then this would cause deletion of the old file.
- ???

2. Add a modified timestamp URL query parameter as described in #146 and #147
Pros:
- Does not alter the thumbnail generation.
- ???
Cons:
- URL query parameters are not always respected by all layers of caching. For example, edge caching in Akamai could ignore the query parameter for some assets such images depending on the configuration in Akamai.
- Depending on the approach to get the modified time, you could be incurring a file system operation ("filemtime") on every image_style theme rendering.
- ???