Strategies for high performance image styles generation
Hi all, i would like open a discussion about high performance image styles generation.
We know that generating image styles some times produce bottleneck, and using s3 buckets the process is more complex and longer, we need to download original image to temporary directory and then manipulate it and push again to the bucket.
Basicly, if there aren't errors, there are three cases:
- Image style not genereted yet and URL is like 's3/files/styles/%image_style'
- Image style generated and URL is like 's3/files/styles/%image_style'
- Image style generated and URL is an Amazon URL with or without CNAME.
In first, Image style not genereted yet and URL is like 's3/files/styles/%image_style', we do all expensive process. Imagine content type with 20 images with 10MB, it's timeout smell.
In second case, Image style generated and URL is like 's3/files/styles/%image_style', your cache stored the intermediate callback, and the problem is that PHP have to appear, sites with a lot of images and demand must support all the process for image styles already generated, Drupal will be bootstraped, then make multiple queries to database and then redirect until expire or invalidate cache. Maybe redirection could be before in the function.
Third case, alright! Amazon URLs don't need Drupal and the performance is better!
I'm thinking different strategies to optimize this. Some ones can be combined.
1. - Make temporal redirections with redirect module. For example: when image will be generated, we add a redirect, the next time if Amazon URLs haven't been generated, redirect module intercept the request and save some of the process. The redirects entries could be deleted with a cron or other process.
2. - Generate image styles in the editor. Use a custom form_submit or hook_entity_update/insert for create a batch to generate only new images added. But the process remains expensive.
3. - Generate image styles in background. When entity have new images, we add the process to a Drupal queue. But until the queue were processed, and cache will be clear the callback 's3/files/styles/%image_style' will receive the requests.
4. - Make a temporary cache (only original files). Duplicate files in the temporary directory with database register in file_managed table with status 0, and create relation table with both FIDs if needed. When a new image style was required, a custom function will search first in temporary directory to avoid download the image for create image style. Finally the cron could be delete temporary file (one day in cache could be enought).
5. - Make a temporary cache (with all files). Generate image styles locally, and only when process have finished, then push to the bucket. If file exists in the bucket show Amazon URL, else show a custom callback that provide the local image.
What do you think about? Any suggestion? The strategies that we talk could be to include in the module like code or like advices in the documentation.
Best regards.
Closed: outdated
3.0
Documentation
Not all content is available!
It's likely this issue predates Contrib.social: some issue and comment data are missing.
Drupal 7 end-of-life triage:
Drupal 7 reached end of life on January 5th.
The 7.x branches of S3FS do not have any additional planned releases.
The requests in this issue do not appear to exist in the 8.x-3.x and newer branches.
Since the issue was created we have cleaned up cache management to invalidate Drupal pages that may have a styles/ link after a style has been generated.