Expose triggering update of media metadata + thumbnail to end users

The Needs Review Queue Bot → tested this issue. It either no longer applies to Drupal core, or fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".

Apart from a re-roll or rebase, this issue may need more work to address feedback in the issue or MR comments. To progress an issue, incorporate this feedback as part of the process of updating the issue. This helps other contributors to know what is outstanding.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Patch for 9.5.3 users.

Patch from #122 doesn't apply for me using composer. I could get it to partly apply using the patch utility but then I had fatal errors.
Here is a new reroll for Drupal 9.5.3

Is there an update hook to install this file? core/modules/media/config/install/system.action.media_update_metadata.yml

For now, I just went through the patch in #123 to identify the reason why the patch doesn't apply.
I found some mix-up in there, namely with Media.php and the media.post_update.php parts. I have taken the gitlab pull request as reference and re-rolled the patch over 9.5.9 version of Drupal core.

Fixed the spelling error in the #126 patch.

Fixed assertion for "Update metadata" operation in test.

Uploading the patch for D9.5.x as well, since #128 fails to apply to that version.

🐛 Remove source from media mappings Needs work may be related as I found when using metadata and trading out an image on a media deletes the files.

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

Re-roll of #129 against Drupal 10.1.x.

Re-roll of #129 against Drupal 10.1.x.

(#134 accidentally included an empty file at a deleted file path.)

#120 is maybe a going concern?

The Needs Review Queue Bot → tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide → to find step-by-step guides for working with issues.

I came across this issue and applied the patch in #134 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work . In general the functionality looks real good and useful. While testing and playing around I've noticed a few details:

1. The status message when running Update metadata seems solely inform the user that the process started and ran, but not about its potential outcomes:

a) You are able to update a media item several times in a row but each time you get the same status message returned: Updated metadata on media item x. Just based on that status message, I would assume that the media item is getting updated with new changes on every run. But if you take a look at the Updated-column on /admin/content/media the timestamp for the media item remains the same and shows an update event in the past. A discrepancy and disconnect between the status message and the Updated-column.

b) After that I was curious and got inspired by #98 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work what would be the status message if you update the metadata for a remote video media type's media item where the URL is not reachable because the server is unavailable/down. Not sure if it is a legitimate way to simulate that scenario, but I've simply switched off the WLAN connection while the site was still running locally in DDEV.
In a Safari window I got You are not connected to the internet for the URL of the remote video's media item. While when I ran Update metadata I've received the same status message "Updated metadata on media item x" and the media item's thumbnail got removed alongside.

c) Not sure how likely it is that a local source file is getting deleted and the Update metadata process is run, but I've manually removed the original image of an image media type's media item. Again I've received the status message Updated metadata on media item x while the thumbnail got removed as in b).

I wonder if instead of informing the user that the process updating the metadata got started, would it be more helpful to provide information about the outcome of that process? That way the user would feel more in control. How about the following rough draft, not perfect at all, just wanted to communicate the gist about each scenario:

• If everything runs through fine with changes in place on the media item's source the current status message could be used.
• For scenario a) how about something like "Nothing to update. Media item x remains unchanged."
• For scenario b) something like "The remote source for media item x is currently unavailable. Refreshing the metadata is not possible, try again later."
• For scenario c) something like "The source file for media item x is missing, unable to refresh the metadata."

2. About the main question if there is a better alternative for describing the action compared to Update metadata? I am uncertain if there is a one or two worded alternative that explains the process clear and unambiguous as a whole. But from my perspective it should be avoided to use loaded terms that might create assumptions, associations, and expectations on the user's end. I think more neutral terms should be used instead preferably. From my perspective Update as well as Metadata are such loaded terms.

Update sounds familiar in the context of updating for example Core or contrib modules but in the context of entities it feels sort of unusual - in particular for locally stored media items in contrast to the remote video ones. In addition to that the term sort of overlaps with Edit one of the other options in the operations drop button. Personally my initial association was either to add something new to a media item and or it might entail additional pro-active manual steps.
Metadata, on the other hand, generates certain expectations. For audio you have for example mp3tags for mp3s, for images and locally hosted videos there is EXIF data and documents like PDFs have extensive metadata too. But if you for example try to create a view based on any of the media items it turns out the metadata is mainly about the thumbnail (in Core out of the box only for remote video and image media types) and the metadata attributes (name, filesize, filemime, width, height, etc) as mentioned in the issue summary. That is totally fine but the term metadata is potentially creating certain assumptions and expectations on the user's end and "might" lead into confusion therefore.

The proposed alternative Refresh (in #10 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work and #60 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work ) for the term Update might be the more clear and neutral choice. From my perspective the term Refresh implies a more unsupervised and automated process.

But in both comments I disagree with the proposed alternative for the term metadata. #10 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work suggests Refresh from Youtube but as @marcoscano points out in #36 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work not every media item is from Youtube therefore that suggestion is not applicable. On the other hand going with the labels suggested in #60 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work I consider those too wordy and it also raises the question what kind information is updated?

Refresh video information
Refresh image information
Refresh local video information
Refresh document information

But I think @devianintegral indirectly provided another viable alternative in #10 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work from my point of view. Why not use the term Source instead of Metadata? The term Source is more neutral but at the same time would imply that you synchronize your media items based on the informations available in the media item's source file. So how about the following changes?

The drop button in the operations column
All options in the select field are single worded, I wonder if it would make sense to stay consistent here?

Edit 
Delete
Update metadata -> Refresh

Bulk action options
For bulk actions I would provide some more context like the other options do and add "from source":

Delete media
Publish media 
Save media
Unpublish media
Update metadata -> Refresh from source

3. As recommended by @benjifisher in #56 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work https://www.drupal.org/docs/8/core/modules/media/creating-and-configurin... → was already added as noted in #57 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work .
But I wonder if it would make sense to create a help topic for the Update metadata task alongside. That way the user wouldn't have to leave the admin ui, plus it would be an easy and convenient way to explain the underlaying functionality as well what it entails aka what data is being updated/refreshed for the different media types.

Those were just my personal observations. I think it would be still good to discuss the issue, in particular point 2 about the microcopy, in a larger group in the weekly usability meeting (as suggested by @benjifisher in #108 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work ).

We've ran into this bug from a slightly different angle with Acquia DAM. We have implemented our own Metadata Sync controller which will attempt to resave the media entity. However the following code in the media entity ensures that if you change the metadata on the external source, it'll never update Drupal because the actual source asset id hasn't changed.

In the meantime we've copied code from core and manually fixed it in our module. Once this issue is fixed, it'll make that code and our metadata sync controller redundant. So +1 to prioritizing a fix here!

Applied #134 succesfully
I can "update" single items but the "update metadata" is not avaliable in the batch selection

The patch in #134 failed for Drupal 10.2 on the media.routing.yml file, so I've adapted the patch slightly.

We had a problema applying #141 following the change of getMetada() into getRawMedata in ✨ Add a hook to modify oEmbed resource data Needs work #59

So here is a patch to apply those changes.

Patch from #142 was not applying because it has been created inside the core folder instead of at the root of the project.
This patch solves that issue and applies on 11.x (and 10.2)

I rerolled MR !3037 on 11.x based on the code from #134 and #141 (next patches were missing some code).

The attached patch applies on Drupal 10.2

Thank you for this feature. Using it on D10.2.2.

For us, one issue remains: browser cache. E.g. updating YouTube video media, new thumbnails will be created but with the same hash (file name). If the browser cache is enabled, the new thumbnail will not be refreshed in the browser.

I'm attaching a patch to fix this issue, using timestamp as a suffix for thumbnail files, and care about removing the old ones on the force update. Moved image_path_flush().

The patch was created from MR3037 diff.

FYI: as @klausi noted, the scanDirectory() can become unusably slow if the directory contains many files.

@szato, we faced the same issue, however we felt changing the logic around the thumbnail name generation might be an overkill and we refrained from solving front-end cache issue with changing the back-end logic.

We have varnish configured on our sites, which further complicates the issue.
So we applied the changedTime of the thumbnail entity to the URI of the image style served, as a get parameter, in an md5 encoded format.
Browser cache relies on the URI too, like varnish does, so cache-busting by get parameter is an acceptable solution which is well separated from the back-end logic of thumbnail generation.
However, we found that the changedTime on the thumbnail entity is not update when we trigger this action, so that might be something to address here!

@joevagyok, if you use hook_preprocess_image_style(), then you can use filemtime($variables['uri']) (the original image file modification time) for changedTime. I think it's better because the entity update time changes every time it's saved again, not just when the file is changed.

Luke.Leber → made their first commit to this issue’s fork.

Rebased against 11.x -- hopefully the test bot doesn't freak out 😅.

Additionally, the concern brought up in #98 is still valid. Upon manual testing, the following procedure results in data loss:

1. Create a media entity that implements oembed
2. Disconnect from the public internet
3. Update the metadata
4. Notice that the original thumbnail is gone and is not replaced with a new one

+1

Related to @japerry's comment in #139, I would note that this approach only handles base field values (string, integer, boolean), and there are cases where it would be important to have source data mapped to, say, a taxonomy term reference field.

The current code can't do that, but I would propose a follow-up issue to make it possible using an event handler.

I can work on that over in ✨ Mapping metadata to reference field Active as a proof-of-concept. Posting here to see if there is an existing issue or indeed, a desire for that kind of feature.

Note: the latest MR 3037 does not have the browser cache busting that was added in #145 ✨ Expose triggering update of media metadata + thumbnail to end users Needs work .

Rebased on 11.x, I added most of #145 except for removing image_path_flush() in Media, that will affect every type not just OEmbed.

@KarlShea
Regarding your comment "I added most of #145 except for removing image_path_flush() in Media, that will affect every type not just OEmbed."
- That seems like an edge case. You would need to have another media type that is using the thumbnail file of an oEmbed resource. The normal usage would be a 1:1 of media to thumbnail file.

Regarding your comment "Edit: actually I removed #145, it looks like there was some disagreement."
There still needs to be a solution for busting cache - browser, edge cache, reverse proxy.

Options so far for cache busting:
1. The thumbnail generation approach as used in patch #145
Pros:
- Busts through all layers of caching - browser, edge cache reverse proxy.
- Automatically deletes the old file and any image styles for the old file.
- ???
Cons:
- A call to "getMetadata" for the "thumbnail_uri" assumes that the thumbnail is being set on the media entity. If it does not update the media, then this would cause deletion of the old file.
- ???

2. Add a modified timestamp URL query parameter as described in #146 and #147
Pros:
- Does not alter the thumbnail generation.
- ???
Cons:
- URL query parameters are not always respected by all layers of caching. For example, edge caching in Akamai could ignore the query parameter for some assets such images depending on the configuration in Akamai.
- Depending on the approach to get the modified time, you could be incurring a file system operation ("filemtime") on every image_style theme rendering.
- ???

Rerolled !3037 MR against 11.x
Patch attached for people needing this on 10.3.1

Hey @DuaelFr applying that patch in 10.3.1 causes update.php to fail.
Specifically: removing the following lines allows the database update to pass:

+    'media_post_update_oembed_loading_attribute' => '11.0.0',
+    'media_post_update_set_blank_iframe_domain_to_null' => '11.0.0',
+    'media_post_update_remove_mappings_targeting_source_field' => '11.0.0',

@queenvictoria Thank you for letting me know! You're right, I did a mistake rerolling this and making the patch. Here is a new one.

Thank you, @DuaelFr for patch #157 for D10.3.

It applied cleanly and works well, but it introduces the following error when drush updb is run:

"The following update is specified as removed in hook_removed_post_updates() but still exists in the code base: media_post_update_oembed_loading_attribute"

I'm getting the same error as banoodle

Oh, @jphelan, you are correct! I don't know why, but I didn't realize 159 is for 10.3 also. Thanks for pointing this out.

Patch for #159 works perfectly for me.

Thanks to everyone for getting this working and to RTBC!

I started triage and immediately saw that the remaining tasks state that a usability review is needed. So, I went through the comments and I do not see a review. Setting back to needs work. I read through nearly all of the comments and noticed some that have not yet been addressed. I have listed those in the 'remaining tasks'.

So, this just needs to work through those items to finish this off.

I've had already added it to our shortlist on 📌 Drupal Usability Meeting 2024-08-23 Needs work . But MR3037 needs a rebase, on a none case sensitive file system ( i am on a mac) i run into the following error:

$> git checkout -b '2983456-expose-triggering-update' --track drupal-2983456/'2983456-expose-triggering-update'
error: The following untracked working tree files would be overwritten by checkout:
	core/modules/system/tests/modules/driver_test/src/Driver/Database/DrivertestMysql/Connection.php
	core/modules/system/tests/modules/driver_test/src/Driver/Database/DrivertestMysql/Install/Tasks.php
	core/modules/system/tests/modules/driver_test/src/Driver/Database/DrivertestMysqlDeprecatedVersion/Connection.php
	core/modules/system/tests/modules/driver_test/src/Driver/Database/DrivertestMysqlDeprecatedVersion/Install/Tasks.php
	core/modules/system/tests/modules/driver_test/src/Driver/Database/DrivertestPgsql/Connection.php
	core/modules/system/tests/modules/driver_test/src/Driver/Database/DrivertestPgsql/Install/Tasks.php
Please move or remove them before you switch branches.
Aborting

it happens when the issue fork was created or last rebased before this issue went in https://git.drupalcode.org/project/drupal/-/commit/8b368d712d83900765744 on the 13th of august fixing spelling errors. the switch to camel case is the problem where the none case sensite file system in combination with git struggles and leads into this error.

Usability review

We first discussed this issue at 📌 Drupal Usability Meeting 2024-09-27 Active . The link to the recording is https://youtu.be/cWtwA63z-IE. For the record, the attendees at the usability meeting were @amazingrando, @anmolgoyal74, @benjifisher, @rkoller, @simohell, and @zetagraph.
We revisited this issue at 📌 Drupal Usability Meeting 2024-10-04 Active . The link to the second recording is https://youtu.be/JPxvcpNXY0Q. For the record, the attendees at the second usability meeting were @AaronMcHale, @avani.bhut, @benjifisher, @rkoller, @shaal, @simohell, and @zetagraph.
Yesterday we revisited this issue for a third time at 📌 Drupal Usability Meeting 2024-11-15 Active . The link to the recording is https://www.youtube.com/watch?v=8DqSpKVK768. For the record, the attendees at the usability meeting were @benjifisher, @rkoller, and @simohell.

After taking a look at the current state of the MR, we’ve mainly focused on the points raised in #138 → , since those cover most of the remaining open questions from an UX perspective, and we’ve assessed their validity and relevance.

1. First, we went through the different scenarios that might happen aside a successful update:

Scenario A (No changes and therefore nothing to update):
It was considered as a nice to have but optional, not a blocker at this point, since it is a difficult task to determine if a media item is changed or not. So we leave it up to the people working on this issue if they want to cover scenario A already within the scope of this issue or if they would prefer to move it to a followup issue. Our consensus for the status message was:

Nothing to update. Media item [x] remains unchanged.

In contrast, the other two scenarios should definitely be covered within the scope of this issue, since in both cases content might be removed and/or lost. As Laura Klein stated in her book “Build Better Products” -

Good design doesn’t lose data or state just because a user has performed an action in a surprising way.

. To prevent any potential data loss, like missing thumbnails in the frontend, the group agreed on adding a confirmation step to each of the two scenarios. So that the user becomes aware, that something is off and that they might experience a potential subsequent data loss, when the metadata of a media item is updated from the source. In the case of remote media items, the source might simply be temporarily unavailable due to an outage, just as, for example, source files where the file folder is being mounted from a file system and that file system is unavailable - without a confirmation step, the data would get removed when the metadata is getting updated. Our recommendation for those two cases are as follows:

Scenario B (The local source of the media item is missing):

Description:
The source file for media item [x] is missing, unable to refresh.

Buttons:
Remove thumbnail
Cancel

Status message after...
…the Remove thumbnailbutton is clicked: Thumbnail removed for media item [x].
…the Cancel-button is clicked: Canceled. Media item [x] remains unchanged.

Scenario C (The remote media item the source is unavailable):

Description:
The remote source for media item [x] is unavailable.

Buttons:
Retry
Cancel

Status message after...
…the Retry button is clicked and the refresh was successful: Updated metadata on media item [x], while if the refresh fails, the confirmation form is shown again.
…the Cancel-button is clicked: Canceled. Media item [x] remains unchanged.

It has to be noted that we had a clear consensus about the importance of having a status message after a button on a confirmation form was pressed. There is always the possibility that someone is clicking a button on a confirmation form, either absent-minded or by a stray mouse click. The subsequent status message keeps the user in the loop and informed about the made choice.

Aside from the refresh of a single source, there is also the case of refreshing more than one source at once via bulk actions. Question is, how to handle things with more than one case of scenario B, more than one case of scenario C, or even a mix of both? You can’t run someone through a chain of tens of single confirmation pages. One option is that the confirmation pages for scenario B might be combined into a single page, the same as the confirmation pages for scenario C, or all confirmation pages for scenario B and C might be combined into a single confirmation page. But due to the fact that there is no existing UI pattern in Drupal Core yet, we don’t want to hold back the current issue, and we would recommend moving the bulk refresh of media items into a follow-up issue.

2. In regard to the microcopy, we considered the term Refresh as suggested in #10 → to be the more clear and accurate choice. Update is a loaded term, in combination with the fact that all the other options (edit, delete, translate) on the Operations drop-down point to forms on other pages. That way, the user’s assumption might be that Update also points to a dedicated page or dialogue that entails an active manual interaction. In addition to that, all the other options in the Operations drop-down are single-worded verbs. Therefore, we would recommend the following change to the Operations drop-down option:

Update metadata -> Refresh

The term metadata is also sort of an abstract and ambiguous term in regard to its scope in this context. It is not clear what kind of data is getting refreshed by the option Update metadata. Is it only the associated fields or also something like the thumbnail, as already mentioned in the issue summary. To make things more explicit we would recommend the following change to the bulk action option label in the follow-up issue (again in line with the suggestion in #10 → ):

Update metadata -> Refresh from source

3. We also checked if there were already help topics for the other options available on the media page, but we were unable to find any. So we had the consensus that it wouldn’t be necessary to add any instructions about the Refresh option to the help topics at this point.

4. One additional detail we’ve noticed during our testing. If you are manually replacing an image in the file system, then hit the Update metadata option, the thumbnail is properly updated on admin/content/media. But if you now click the Edit-button for the media item and then click the link to the source file, the old replaced image is still shown. Could that be a caching issue? We haven’t had the time to investigate in more detail.

I am setting the issue back to Needs work and remove the Needs usability review tag. If you want more feedback from the usability team, a good way to reach out is in the #ux channel in Slack.

Wondering if this improvement will also work for images stored in cloud storage options such as S3, Azure, etc. via modules such as the S3 File System → and Flysytem → ?
Once the images are stored in the external cloud storage, will this still work in capturing exif data on demand for already loaded/saved images and media?