Why is an Identical Workflow Requiring More Memory to Run?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hi team :wave:

I had an unexpected Out of Memory error on one of my workflows yesterday. I tried re-running it with more memory, I went from initially 512MB to 1024MB and then the maximum value 10240MB. But every rerun failed with the same Out of Memory error.

The worflow in question is fairly complex (8 steps in javascript, bash and python) but there is no way 10GB of memory isn’t enough to run it.

I have an identical workflow that I use as a testing environment and it was able to run the same request with 512MB of memory.

Any idea what could be causing this?

Thanks a lot :pray:

2

I’ve had other requests coming on this workflow afterwards and they completed without Out of Memory issues.

3

I think this might related to the amount of data that your workflow process? Maybe the workflow in your testing environment is not handling the real data?

4

Yes you’re right the testing workflow processes far less data that the production. But isn’t the memory emptied between requests? I’m sure that one request does not need more that 1GB of memory to run. So it would mean that it is an accumulation of previous requests memory that triggers the overflow :thinking_face:

5

Hi , I think the memory is not accumulate between the runs. May I ask if the error happens for every run, or is it happen for just some events?

I suspect that only some specific events caused this issue.

6

It happened for a specific event, it tried re-running it 3 times with increased memory but it always failed. However, I tried it again this morning (same event) and this time no memory error.

7

I think this might really related to the logic in your workflow. Could you check if is there any step that fetch more data, duplicating the data, or transforming the data? Those operations might incur more memory

8

There is a step that download a file and then another one that stream this file to an external API. So it makes sense that this workflow needs a fair amount of memory to run but it should be nowhere near 1GB or 10GB. Also I don’t understand why it works this morning if it failed yesterday.

9

And there is something else that’s surprising. In the error details, it says the workflow ran for almost one minute (Compute Time: 57,759 ms) but went out of memory before running any step (Steps Executed: 0 / 8 ). How is that possible?

10

I think the Steps Executed 0/8 is an UI issue. actually it failed at one step.

> There is a step that download a file and then another one that stream this file to an external API
Are you able to get the average size of the files the workflow download?
I think this would multiplies the memory needed for your workflow. As it would need to store all the file content into memory. I think if you can offload this process somewhere then the problem should be fixed

11

It’s hard to tell the average size of files but it is rather small. For example the size of the file responsible for yesterday’s memory error is 13MB. So the initial 512MB of memory should have been enough.

12

I could offload this process to an AWS EC2 or Lambda but that beats the point of using Pipedream :grimacing:

13

I see. Maybe or @UMT4G7E5P can help you in this case when he comes online in a few hours!

14

Yes hopefully! Thanks for taking the time to help :pray:

15

in the mean time, is it possible for you to share your workflow url?

16

I am also having issues with out-of-memory errors on a number of different workflows. These are workflows that have been working fine for a while, but suddenly running into issues lately (i.e. in the past few weeks). Sometimes I can re-run an event without making any changes to my workflow and it will work the second time, though sometimes it continues to fail.

In some cases my workflows are handling image files, but I am streaming these files to disk so I don’t think it is the issue. Again, the workflow used to handle large images just fine. Plus not all the records I’m processing involve image files, and even records with no images still sometimes throw an out-of-memory error.