This topic was automatically generated from Slack. You can find the original thread here.
Hello there!
I wonder if there is a possibility to make pauses on each “for” cycle iteration inside the code step, that way it does not cause TIMEOUT WF error.
I mean, I do know that I can set up “Execution Controls - Timeout” to 750 seconds, but it may be not enough in some cases.
For instance, I may have 800 API requests, and I need to send them all inside 1 code step. However, I am not able to send them all at once, coz I have API limits.
So, I need to have the opportunity to implement a specific time gap between each cycle iterations.
However, I’m certainly going to hit total WF time limits, if I use default Node.js tools.
The workaround, which I do use now, is to send post requests to the other WF, which is executing the API requests. This other WF have specific throttling settings.
However, there are 2 issues. Firstly, I don’t wanna use 2 WFs for one process. And, which is more important, If send too many PD → PD POST requests, I get a TIMEOUT error as well.
Hope you got my point. I wonder if there are any solutions. Also, it seems to me that $.flow.delay only works once after the step’s ending.
Thanks for your answer in adwance!
Hi, I think your question is a great case for Pipedream bot to help. I’ll pass your question to the Pipedream bot now. Please feel free to ask it any question
I wonder if there is a possibility to make pauses on each “for” cycle iteration inside the code step, that way it does not cause TIMEOUT WF error.
I mean, I do know that I can set up “Execution Controls - Timeout” to 750 seconds, but it may be not enough in some cases.
For instance, I may have 800 API requests, and I need to send them all inside 1 code step. However, I am not able to send them all at once, coz I have API limits.
So, I need to have the opportunity to implement a specific time gap between each cycle iterations.
However, I’m certainly going to hit total WF time limits, if I use default Node.js tools.
The workaround, which I do use now, is to send post requests to the other WF, which is executing the API requests. This other WF have specific throttling settings.
However, there are 2 issues. Firstly, I don’t wanna use 2 WFs for one process. And, which is more important, If send too many PD → PD POST requests, I get a TIMEOUT error as well.
Hope you got my point. I wonder if there are any solutions. Also, it seems to me that $.flow.delay only works once after the step’s ending.
I understand your concern about making pauses between iterations in a “for” loop to avoid hitting API limits and workflow timeouts. Unfortunately, using $.flow.delay or other delay methods within the loop will still count towards the total workflow execution time, which might cause a TIMEOUT error. One possible solution is to use the built-in Delay app to create a separate workflow that handles the API requests with a delay between each request. You can trigger this workflow from your main workflow using an HTTP action or by using the axios library to send POST requests. This way, you can control the delay between requests and avoid hitting API limits or workflow timeouts. Here’s an example of how to use the Delay app in a separate workflow: 1. Create a new workflow with an HTTP trigger. 2. Add a “Delay Workflow” action from the Delay app. 3. Set the desired delay duration (value and unit) for the action. 4. Add a code step or an action to make the API request. 5. In your main workflow, use an HTTP action or axios to send POST requests to the new workflow’s HTTP trigger URL. Remember that this solution still involves two workflows, but it should help you manage the delay between requests more effectively.
That’s the feature I figured out and use as well.
However, I’ve faced a case, when I had to send 1700~ requests to a Workflow-executor (processing wf), and it caused TIMEOUT error in a main (first) WF.
I suggest taking a look at building a source specifically.
Then you can paginate this CRM’s search results and scale the workflow by batching the processing into single workflow executions on one record at a time.
And which CRM are you trying to sync data from? Pipedrive, Hubspot or SalesForce?
It’s SalesMate for us. And thanks, I will take a look.
I think that I may try a buffer-database as well, when I need to have all the data to be stored in a specific place at once. Then, I might try to use this BD to operate with the data from different WF’s executions.
