Why does my workflow execution time vary for different events?

Help
,
1

This topic was automatically generated from Slack. You can find the original thread here.

Andre Brunetta : Hi, I’m experiencing a wide range of response times, from 1.5s to 18s without changing any code and also using the same payload (replay). Just wanted to make sure I’m not doing anything wrong here… is that normal / expected?

2

Andre Brunetta :

image.png
image.png388×614 14.5 KB

3

Dylan Sather (Pipedream) : Hi , are you making requests to third party APIs within the workflow? A lot of the difference in runtimes we observe are due to differences in the response time of HTTP requests. Is it possible a service you’re using simply took a long time to respond in this case?

4

Andre Brunetta : Hey there are requests to third party APIs but testing outside the workflow they apparently are responding consistently quick…

5

Dylan Sather (Pipedream) : if you replay the event once more, do you observe the same latency?

6

Andre Brunetta : these were the last replays:

image.png

7

Dylan Sather (Pipedream) : so it looks like the issue could have been intermittent?

8

Andre Brunetta : Yes, I’d say so…

9

Andre Brunetta : A few replays more and it happened again:

image.png

10

Dylan Sather (Pipedream) : Pipedream workflows are subject to “cold starts”, but that time shouldn’t be reflected in the total time you see here. In general, we do observe a large distribution of response times from APIs, and that is also expected. e.g. I have a workflow that talks to Slack and most requests take less than 1s to complete, but some take upwards of 20 seconds, and some even time out (for the same API call). This is just the nature of third party APIs.

11

Dylan Sather (Pipedream) : If you’d like, you can add step-level timing within each step, to assess what specifically is taking the largest sum of time:

this.start = (new Date()).toISOString()
// your code
// ...
// 
this.end = (new Date()).toISOString()
12

Dylan Sather (Pipedream) : that’ll provide start and end step exports below the step

13

Dylan Sather (Pipedream) : if you observe anything odd there (i.e. something that’s not explained by a request to a third party API taking a long time), feel free to share your workflow with dylan@pipedream.com and I’m happy to dig in

14

Andre Brunetta : Great, thanks again Dylan! I’ll try to investigate those times and will let you know if anything pops up

15

Andre Brunetta : Hey , in the following example, this.duration is only 1. Perhaps it’s not waiting for the $sql.send execution?

this.start = new Date().getTime();

await $send.sql({ 
  table: params.table,
  payload: payload,
});

this.end = new Date().getTime();
this.duration = this.end - this.start;
16

Dylan Sather (Pipedream) : Believe it or not, that’s expected and by design! Destination delivery happens asynchronously, outside of your workflow, to keep the execution of your workflow fast.

17

Dylan Sather (Pipedream) : That’s most impactful for sending HTTP requests (when you use $send.http() - https://pipedream.com/docs/destinations/http/#using-send-http). When you don’t need to access the HTTP response within your workflow, you can send the request to the service async using $send.http(), so your workflow won’t wait on the HTTP response.

18

Andre Brunetta : Great, thanks!