What Are the Limits for Timeout Controls and How Can I See Which Steps Take the Most Time?

1

This topic was automatically generated from Slack. You can find the original thread here.

Hey, I have two related questions

  1. What is the limit I can set for timeout controls? I have a large workflow which is already taking close to 30s on a cold start and want to prepare for it
  2. Is there a way for me to see which specific steps are taking the most time? I’d like to maybe optimize specific steps of the workflow

image.png

2

> 1. What is the limit I can set for timeout controls? I have a large workflow which is already taking close to 30s on a cold start and want to prepare for it
Hello , you can refer to the doc here for more detail on the Timeout limit. I paste the important part below

You can increase the timeout limit, up to a max value set by your plan:
TierMaximum time per execution:
• Free tiers: 300 seconds (5 min)
• Paid tiers: 750 seconds (12.5 min)
Events that trigger a Timeout error will appear in red in the Inspector. You’ll see the timeout error, also in red, in the cell at which the code timed out.

3

thanks, does it use more invocations if it goes longer than 30 seconds?

4

Yes a base credit is 30 seconds of compute time per 256 of memory.

https://pipedream.com/docs/pricing/#credits

5

technically on your current plan, extra compute time doesn’t count more invocations. We changed to a credits-based model for new users because we bear the cost of the compute. But for now, no additional cost beyond the first invocation (unless you raise the memory of the workflow, as well)

6

I ran a Workflow A that was dumping single bits of data (a barcode) into Workflow B that was looking them up on another API. Workflow B was limited to run 1/s because of the API rate limit on the lookup. Workflow A has 1200 entries to send so it was timing out after 12.5 minutes. Why is Workflow A not able to simply dump the data into Workflow B’s queue (set to 2000)? Why does it have to wait for B to process the item before sending another?

7

Does Workflow A wait on Workflow B to return a custom HTTP response?

8

Can you show me the configuration of the HTTP trigger in workflow B?

9

Sure…

10

I guess the issue is that the rate limit of 1/s applies to the GET request going out and the RECEIVED message going back to workflow A? I used the settings on Workflow B to limit it to 1/s.

11

image.png
image.png408×678 19.6 KB

12

Based on these configuration settings, workflow A should be dumping requests into workflow B without waiting for workflow B to finish. Since you’ve set workflow B to issue a static HTTP response with a 200 OK, we should accept the request, enqueue it, and return a 200 OK immediately (independent of workflow B’s configuration).

Can you visit the workflow Settings for workflow A and share the workflow with Support, then share that URL with me via DM? I’d like to see exactly how that workflow is configured to send requests to workflow B, to see if anything else explains the long runtime.

13

Will do. When I was testing during setup I got a “429 statusText: Too Many Requests” so I assumed sending 1200 at once had killed the webhook at Workflow B (it received approx 950 before dying). So I slowed it to 1/s to match and then realised that was going to cause timeout issues. I then changed it to something like 5/s so that it would not crash the webhook and fit under the timeout limit. I realised after deploying that whatever I did it was processing 1/s on Workflow A (I counted!) so I removed all the settings limits to try and get it across as fast as possible:

image.png
image.png606×644 19.7 KB

14

we can increase the QPS of the HTTP endpoint for workflow B so that it can accept larger bursts of traffic. Can you send me a DM with the workflow’s HTTP endpoint?