Optimize drush command

I was about to create a similar issue but I found this one. I have ~48,000 files and the Drush command never manages to reach till the end. Hopefully, the solution to this issue will solve my problem.

To avoid having a huge number of items inserted into a batch, it might be more efficient to store a highwater mark. For example:

* Batch starts with "position = 0"
* Load, say, 50 file entities where fid > 0.
* Process the files and after processing each file, set the position to "fid" for that file.
* Keep repeating 1 and 2 until step 2 returns an empty result, which means all files are processed.

This approach should be able to tackle any number of records without spending additional resources to create an initial "list of tasks".

PS: I haven't had the opportunity to see the open pull request yet.

Updated the tests and added the two missing features from summary:

• Option to order loading of files by created date (default: DESC = newest first).
• Only load permanent files from database.

Reverted the changes to logging made by @DieterHolvoet, I think in overall the logging would need a bit more sophisticated approach, but I'd leave it as a follow-up issue.

We also should try to add new Drush command options to PHPUnit testing or open a related issue for this.

sokru → changed the visibility of the branch image_style_warmer-3204351-3204351-optimize-drush-command to hidden.

Added few tests on this MR !7.

Should the README also be updated to include an overview of the drush command, especially since it's being enhanced here?

@mikeohara: Sounds valid to me.

I think in the near future Image Style Warmer should also get some Gitlab doc pages.

Hi
I try to use all you talk about here, but where is the doc to know how to use drush with ID asc and other options ?
I don't see anything in hte readme or elsewhere

thank you

Just made some commits to fix the pipeline.