How to Diagnose Out Of Memory Errors at an Endpoint in a Workflow Despite Increasing Memory and Having a Non-Large Payload?

Help
21

It’s 12:10 now so in 2 hours

22

I’m looking at your workflow and the public Google Cloud components import quite a few npm packages:

"@google-cloud/logging"
"@google-cloud/storage"
"@google-cloud/bigquery"
"@google-cloud/compute"
"@google-cloud/bigquery-data-transfer"
23

Each step imports all the packages it uses, so having multiple steps that import many or large packages may result in an OOM error

24

You mentioned the workflow was running okay before — was there a change or were steps added in the last deployment?

25

Hmm :thinking_face: I set both of my workflows to 768 and we’ll see, there exactly the same except one runs at 9AM and the other at 2:00PM and they find different videos in that spreadsheet is the only difference.

26

If you want to see the other one

https://pipedream.com/@weirdorfunnyfacts/projects/proj_zNsq4pq/youtube-sunrise-trivia-p_MOCaZ9M/inspect

27

This one the max video size is much smaller around 20 MB though.

28

Hmm yeah looks like the exact same steps yeah?

29

Is there a way to pass the public URL for the GD video instead of downloading it to /tmp?

30

Hmm :thinking_face: and just have YouTube pull it directly from Google Drive :thinking_face:

31

Yeah, I added in a couple more Google Cloud steps. I guess I meant that the trigger was working fine, and now it wasn’t.

Is there any workaround to the run this where the packages are only imported once? Like can I do these steps all in a Code step for example?

32

I’ll mess around with it and see if that would work

33

Yeah, I added in a couple more Google Cloud steps. I guess I meant that the trigger was working fine, and now it wasn’t.
Oh what happened to the trigger?

34

Is there any workaround to the run this where the packages are only imported once? Like can I do these steps all in a Code step for example?
Yes, that would be a workaround

35

I am just speaking poorly today…

From my view the trigger was what as causing the OOM error.

36

Ah yeah, np that is what it looks like

37

But you have now clarified that it is the steps and the loading of the relevant libraries.

38

Yeah exactly!

39

Can “Run Query” step run multiple queries? eg. separated by a “;”?

40

Not currently, but it makes sense to me we could build an action that runs multiple queries