Add support for s3fs to use the assets:// stream wrapper

This has caused issues when running Drupal in a stateless infrastructure such as Kubernetes. It was nice delegating the Drupal files directory to S3 so that there isn't a need to setup a persistent volume with a shared EFS for each pod instance to mount.

Now that Drupal 10.1 has moved these generated files to `assets://` it puts more onus on developers to manage the sharing of these generated assets when scaling infrastructure horizontally.

The storage of CSS/JS on assets:// is currently stateless safe.

CSS/JS content is now only generated on request, not on each page generation. A request link includes instructions to rebuild the file from local disk.

In a horizontally scaled environment this would be content a caching load balancer(or CDN) could handle without need to mount shared storage on each container. One could also use a dedicated container for asset generation.

Add a `s3fs_use_s3_for_assets` configuration to the module that allows you to push these assets to S3.

While this would be the end user side of this, actual backend work is much more complicated. See 🐛 Breaking changes for public:// takeover in D10.1 from #1014086 Closed: outdated . This would tie us very tightly to core API and is something I would like us to avoid if possible.

The assets:// wrapper was cores response to a BC break in 10.1, technically itself the feature is arguably a BC break too, however with BC breached it was cores method to reduce the impact.

@tom friedhof

Do you still see this as a necessary feature given the context from #2 regarding the assets:// StreamWrapper not requiring a stateful filesystem?

@cmlara It wasn't working the last time I tried. For now I have left aggregation turned off until I have chance to take a closer look at why assets are not being generated on request, as mentioned in #2

I should have time to pick this back up at the start of the new year. Thanks checking in!

Sounds good, I’m going to set this postponed until after the new year.

I will add a bit of advice:
Ensure your web server is configured for the new path. Nginix is known that most sites will need a config update and it isn’t guaranteed other server types will not need adjustments as well. More details are in the change record https://www.drupal.org/node/2888767#nginx-php-fpm →

Additionally validate any modules you have that manipulate CSS/JS aggregation are still compatible. Advagg for example is known, as of this writing, to still not be compatible with D10.1+ due to the core changes.

Hi, is this finally going to be addressed within the context of S3FS? I am not sure if it is intended to allow s3fs to be able to somehow manage assets again with the new stream wrapper. If that possibility exists within the module roadmap, it might be interesting to bring this issue back to active status to see if we can push the functionality forward, either by the maintainers or the community itself.

I know that there is a module that provides this functionality, but I am not sure if it is the best approach, and on the other hand if all this would be within the context of S3fs it would be more interesting.

Translated with DeepL.com (free version)

@Juanjol
This is not currently planned to be implemented. This issue was left open to allow feedback to validate that it is no longer needed in a multi host cluster now that Drupal 10.1 is available.

Can you advise why you feel s3fs needs to support storing the assets:// streamWrapper? What are you trying to accomplish?

@tom friedhof
Did you ever get a chance to confirm the horizontal scaling no longer requires s3fs to handle CSS/JS aggregation in the latest Drupal Core builds?

I never got this working properly with s3fs and assets. I ended up switching to using Elastic File System and mounting a volume to the pods.

I never got this working properly with s3fs and assets. I ended up switching to using Elastic File System and mounting a volume to the pods.

I'm curious what issue you encountered?

I just finished rolling out a new local dev cluster this week so I have some spare resources to do a quick test on this:

1. Based off the Bitnami Drupal helm chart, configured for RWX share on the files (this shares sites and module directories)
2. Manually install s3fs on each node with composer (to populate vendor folder). Technically I should have built a custom docker image and ran composer inside it, however this was just a quick test.
3. Configure s3fs settings in settings.php including public:// takeover.
4. Set $settings['file_assets_path'] = "assets2"; in settings.php (changing path is because the sites/default folder is shared in the Bitnami chart, however I did validate that without this setting url's were correct to sites/default/files/ which ensures that public:// takeover was not interfering with URL generation.
5. Clear cache on every node ensure setting is picked up (this could be done with a deployment identifier, however i just drush cr on both test nodes through remote shells)
6. Navigate to http://DrupalHostLBIp/
7. Observe page loads correct, assets served directly from the Drupal cluster nodes. Observing pod logs it is clear that the CSS was generated by multiple hosts in a load balanced manner. Disabling browser cache and refreshing the page multiple times caused both pods to eventually each have have duplicate copies of the CSS/JS files
8. As noted above a more complex deployment could be configure to route 100% of CSS/JS generation traffic to a specific pod if desired instead of spreading the load across several pods.

I have been informed that a ecosystem module s3fs_assets → exists to fill this role.

While I am not sure that module as it currently exists would be a good fit for all sites as currently written improvements can be handled in s3fs_assets queue.

I'm inclined to mark this issue won't-fix and allow the ecosystem to handle it.

I will leave this open for 7 days to allow any objections to be raised.

Closing as won’t-fix to allow s3fs_assets → to handle this.

I will again note for most sites this likely is not needed as assets:// is currently stateless safe (every server can re-create the missing files).