How to Diagnose Out Of Memory Errors at an Endpoint in a Workflow Despite Increasing Memory and Having a Non-Large Payload?

Help
41

Yeah, I am just trying to think how I can simplify the workflow.

42

Would you mind requesting that in our public repo?

43

Yes, I can do that.

44

Cool, instead of a string input prop it can be an array of strings

45

I believe that should not be an issue with the way the action is built currently

46

Sorry, just to clarify, it can be an array now? Or that would be a potential implementation to this feature request?

47

Oh sorry I meant it would be a potential implementation

48

Btw, are you familiar with publishing private actions?

49

And customizing the public actions/triggers

50

I mean this guide: Pipedream Registry - Pipedream
In step 2, instead of creating a new component you can edit an existing component and then use the CLI to publish it privately to your workspace

51 
pd publish <filename>

CLI Reference

52

I am not sure that is within my skillset :wink: I could try and muddle through that though.

53

No worries! Just throwing options

54

Hi I reduced my workflow, using a code step, to 1 Google Cloud step + importing just BigQuery into my code step itself and it is still throwing the OOM at 256MB.

I upped it to 512MB and it seems to complete more reliably, but this uses 2 credits per run. Cost-wise I feel that isn’t going to be sustainable as this will definitely have overages.

Any other workarounds you can think of to reduce the memory footprint? 512MB to run a few text manipulations and queries seems like a massive overhead.
I left it open to inspect if you’d like

55

It doesnt look like its possible to directly use a google drive link to upload to youtube without downloading the video first.

56

Using a direct link google drive download its not able to complete the task.

57

Actually, I removed the last of the Google Cloud steps, and replaced it with the same code step, with different input and it seems to be working. So I think it is because the Google Cloud steps import more than needed. I will keep an eye on it, but so far so good.

58

Damn, spoke to soon… They have started failing again. And now there are no Google Cloud steps, just imports on two of my code steps… :disappointed:

59

I see a few runs that failed with OOM, but the most recent ones did run successfully (except last 2, other error)