Executing from Code

Created on 28 October 2024, about 2 months ago

I have to say that I think this is a great module. Speaking as someone who's built more custom batch jobs than I care to remember, I can attest to the pain and agony it entails. Batch API is both one of Drupal's most useful and yet most difficult tools.

Problem/Motivation

That said, I would love to have a way to execute these batches from code and/or integration with Ultimate Cron.

I asked you about this in another ticket and you provided a workaround to do this, but you then said...

"However, this is not a true batch. as it will not invoke the batch api and progress bar with repeated ajax calls. This means it is subject to timeouts since it is just one process running until it timesout or finishes, whichever comes first."

Which basically negates the entire purpose of using this module.

Proposed resolution

Actual integration with Ultimate Cron isn't really necessary, I can create an empty Ultimate Cron hook easily enough. If I have to way to fire off a Codit batch from inside that hook then we're there. Your Cron implementation is slick, but not nearly granular enough.

API changes

Data model changes

✨ Feature request
Status

Active

Version

1.0

Component

Code

Created by

πŸ‡ΊπŸ‡ΈUnited States cdesautels

Live updates comments and jobs are added and updated live.
Sign in to follow issues

Comments & Activities

  • Issue created by @cdesautels
  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    I'd be interested in hearing the granularity that you need from ultimate cron. It might be easier for me to replicate the granularity you need than try to make a way that a single function call invoke an ajax based batch.

  • πŸ‡ΊπŸ‡ΈUnited States cdesautels

    Sure. All Ultimate Cron does is find every cron_hook in the system and allows you to set specific Cron rules for it. Then the best practice is to set Cron to run VERY frequently, but it only runs the tasks the Ultimate Cron's config says it should. So you have have tasks the run once a year or every minute, or anything in between that is possible with Crontabs rather flexible rules (much more flexible then your set up). You can know exactly when a given task will run and because you can create your own Ultimate Cron hooks you can even apply conditions in code that allow a given task to run or not run, schedule be dammed.

    Your rules are rather vague, "on the 1st of every 1 month". What does that mean? Midnight? 3am? noon?

    So for Codit. I suppose one could put conditional logic in the gatherItemsToProcess() method to override a schedule. But it would be nice if there were a way to fire a batch directly from code for some edge cases I've come across.

    A minimum viable change would be to provide a means to enter an explicit Crontab rule rather then the plain english string you currently support. I imagine that that string is being converted to a crontab rule somewhere, so it should be too hard to proved a crontab rule option.

  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    The rules are actually pretty precise, only as vague/precise as the frequency that system cron it set to run.
    "on the 1st of every 1 month" precisely means it will run the first time cron fires on the first day of every month. So if your cron is set to run every 10 minutes, it would run at 12:10AM on Nov 1st, Dec 1st. If your cron is set to run every 1 minute, it would run at 12:01AM. Ultimate cron suffers from the same lack of precision on timing. It is only as precise as how often cron is set to run, which is why they suggest setting cron to run on small intervals (more often).

    If you need to be more specific about the time of day, the pattern supports it
    "on the 1st of every 1 month after 8:22" would run on the first cron run to fire after 8:22AM.

    This module does not use crontab rules under the hood, because there are too many scenarios that crontab rules can not support.

    Something as simple as "every 5 months" (5 is not a factor of 12) or "every 3 days" (3 is not a factor of 7) or "every 2 years" (crontab has no concept of years) can not be accomplished with crontab rules.

    Conversely, I can only think up two crontab rules that the CBO patterns can not reproduce. I don't think they would be hard to add, I just have not had reason to prioritize them yet.

    1. ✨ New time pattern request: At the x:NN Active
    2. ✨ New cron time pattern request: Days of the week. Active

    I put a pretty serious amount of thought into this when I built Codit:Crons β†’ for D7. It was largely the predecessor for CBO's cron logic.

    So the question is, what cron pattern would you need if you could use crontab rules? I am pretty sure I can tell you what that would look like in english.

  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    Thinking about your situation some more. I think calling the method as I described for you earlier would actually work fine if called from an ultimate_cron hook. The reason why it is a pseudo batch when called directly is that it can not instantiate an ajax connection that keeps recalling the batch until it is done. Something running by being called in any kind of cron hook would never be able to instantiate an ajax connection because it is never running in a browser. When it is called, cron has invoked it. And I think... semi-sure?... that cron runs essentially like CLI, and is not subject to browser connection timeouts. It would be similar to running it with a Drush command.

  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    ehhh So I found that it is not completely boundless.
    https://www.drupal.org/docs/7/setting-up-cron/troubleshooting-cron#:~:te... β†’ .

    All crons combined can not exceed 240 seconds.

    However, this would not be an issue because of the way BatchOps keep state and the way it also tracks completion of a cron.
    Example: Lets say you had 10,000 nodes to process, and in the first 240 seconds it only got through 8,701 before the process timed out. The run would have recorded that it got finished 8,701 but would not have recorded the cron was finished. So the next invocation of cron, let's say 5 min later, it would pick up at 8,702 and keep going. Only after it completed 10,000 would the cron run have recorded itself as being done. All that is baked in.

    I don't think there is a better solution available other than having an external cron runner that operates outside of drupal.

  • πŸ‡ΊπŸ‡ΈUnited States cdesautels

    Ok, sounds like you gave a lot of thought to the cron integration. I think you probably should explain this in more detail in the docs and provide some examples of the more arcane cron strings.

    As for the code invocation hack. I don't currently have a scenario where I can test it. But probably will in the future. If at that time, I find a problem, I'll be in touch.

    Thanks for your prompt reply.

  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida

    Thanks for the recommendation of better documentation. I am feeling like most of what I said above is covered in the Readme / Project page :

    Here is the bit that describes the pattern

    Running a Batch Operation via cron
    When coding your batch operation, define a public function getCronTiming() and have it return an acceptable pattern. The patterns used are human readable. Example:

    'every 10 minutes'
    'every 2 days'
    'every 2 weeks
    'every 3 months'
    'every 2 years'
    'every 10 days after 16:00'
    'on the 6th of every 1 month'
    'on the 15th of every 1 months after 16:00'
    'on the 4th of July'
    'on the 4th of July after 14:00'

    And this bit describes more about the timing

    Why do the cron time settings say 'after' instead of 'at'? Cron timings on cron jobs from this module are evaluated every time cron runs. Due to the settings of your cron, you may have a cron that only runs every 2 hours, so the timings of the cron job can only be tested every 2 hours. If you want your times to be more accurate, set cron to run more often (warning this can have performance issues).

    If you have ideas on how that can be improved issues with patches are always welcome.

  • πŸ‡ΊπŸ‡ΈUnited States swirt Florida
Production build 0.71.5 2024