Pause a Workflow for the Rest of the Day

Help
1

I’ve put together a workflow that scrapes a website every 5 minutes between 8:00A and 1:00P looking for information that may or may not change. I’ve done the timing with a cron block as the first block in the workflow, and the workflow has been working flawlessly for the last year. Now I am wondering if I there is a way that I can be more efficient with the workflow. More specifically, if the workflow finds that the information has changed at 10:00A, there is no need for the workflow to continue for the rest of that day (though it does need to resume the following day). So, my question to the experts is: is there a way to set a flag that overrides (pauses) the cron block if the information was found but can somehow be reset at the end of the day?

2

Hi @dplescia,

Yes, you can achieve this by using the $.flow.exit() method to stop the workflow execution for the rest of the day if the information has changed. You can also use a cron expression to reset the flag at the end of the day. Here’s an example of how you can do this:

  1. Add a variable to store the flag, for example, informationChanged:
let informationChanged = false;
  1. In your workflow, after you check if the information has changed, set the informationChanged flag to true:
if (/* information has changed */) {
  informationChanged = true;
}
  1. Before the main logic of your workflow, check if the informationChanged flag is true. If it is, use $.flow.exit() to stop the workflow execution:
if (informationChanged) {
  $.flow.exit("Information has changed, stopping the workflow for today.");
}
  1. To reset the flag at the end of the day, you can create another workflow with a Schedule trigger that runs at a specific time (e.g., 1:00 PM) and resets the informationChanged flag to false:
informationChanged = false;

Remember to set the Schedule trigger for this new workflow to run at the desired time, like 1:00 PM.

This way, your main workflow will stop executing for the rest of the day once the information has changed, and the flag will be reset at the end of the day by the second workflow.
I’m a bot powered by Pipedream and GPT-4. I’m still learning, so please double check my answers and code! Learn how to build your own.

3

Interesting idea,but I am not seeing how this pauses the cron for the rest of the day. It seems like the flag will still have to be evaluated at each tick of the cron - the remainder of the workflow simply won’t execute. Am I missing something?

4

There might be a way, but it would have to be done within one single Node.js code step.

Using $.flow.rerun, you could check if your condition is met, and otherwise keep waiting until it is.

// Re-run every 5 minutes, stop retrying after 60 attempts (for 5 hours, from 8AM to 1PM).
$.flow.rerun(1000 * 60 * 5, null, 60);

This way, you would only need to schedule your workflow to run once per day, rather than every 5 minutes. The workflow/code step would then pause & resume/retry every 5 minutes, until your condition is met and you make the step return successfully instead (and continue with the rest of the workflow).

Does that make sense?

3 Likes
5

Makes perfect sense! Many thanks!