This topic was automatically generated from Slack. You can find the original thread here.
What does this mean?
Message:You've exceeded the default memory for this workflow. You can raise the memory in your workflow's Settings. See https://pipedream.com/docs/troubleshooting/#out-of-memoryTotal Duration:50,766 msCompute Time:255,000 msExecution Start:2023-10-06T13:46:03.783ZExecution End:2023-10-06T13:46:54.549ZSteps Executed:1 / 17Credits:3Version:87 (d_75skm29k)
This message indicates that your workflow has exceeded the default memory limit. To resolve this issue, you can increase the memory allocated to your workflow in its Settings. For more information, please visit the troubleshooting guide on Out of Memory errors: Troubleshooting Common Issues
@U02A06JPKEH @U02SX7ETBFB
I have increased the memory to 375.
The workflow works 25% of the time and is critical to my business.
Can anyone help me with this?
can you describe the various reasons why a workflow would run out of memory and how to resolve them?
A workflow can run out of memory for two main reasons: 1. Loading a large file or object into the workflow’s memory (e.g., saving the content in a variable). To resolve this, consider streaming the file to/from disk instead of storing it in memory, using a technique like this: https://pipedream.com/code/nodejs/http-requests/#download-a-file-to-the-tmp-directory. 2. Having many steps in your Pipedream workflow. Each step in your workflow runs as a separate process, which incurs some memory overhead. This typically happens when you have more than 8-10 steps. To resolve this, try increasing the memory allocated to your workflow in its Settings. For more information, please visit the troubleshooting guide on Out of Memory errors: Troubleshooting Common Issues
If you need help with a specific workflow’s implementation, you can open a support ticket here: https://pipedream.com/support
This link doesn’t work.
1. Loading a large file or object into the workflow's memory (e.g., saving the content in a variable). To resolve this, consider streaming the file to/from disk instead of storing it in memory, using a technique like this: https://pipedream.com/code/nodejs/http-requests/#download-a-file-to-the-tmp-directory.
Here’s the good link: Working with the filesystem in Node.js
You can also do it like this (using the native fetch feature), which I personally find better:
Nice, thanks for sharing Marco!
I’ll get that link fixed as well, sorry about that.
and @U02SX7ETBFB (@U05A7KLNALC)
One small observation that I’ve noticed is that when I run the workflow by hand, it works 100% of the time.
This workflow is trigger by a workflow in ASANA.
It passes the taskGid variable thru HTTP and runs the workflow.
It’s interesting that the workflow works 100% of the time when you run it manually but not when it’s triggered by an Asana event. There could be a few reasons for this behavior: 1. The Asana event might be sending different data or in a different format than what you’re testing manually. Double-check the data being sent by the Asana event and ensure it matches the expected format in your workflow. 2. There might be a race condition or timing issue when the workflow is triggered by the Asana event. You can try adding some delay or retry logic in your workflow to handle any potential timing issues. 3. The Asana API might have rate limits or other restrictions that could affect the workflow when it’s triggered frequently. Check the Asana API documentation for any limitations and adjust your workflow accordingly. If you’re still having issues, please visit https://pipedream.com/support for more support options.
(@U05LWJMUX0A @U02SX7ETBFB)
I set up a workflow in Asana, to trigger a Outgoing GET request.
The instructions read as follows:
A call will be made to this endpoint using a GET request, and the task gid will be passed as a query param called "taskGid". Email us at [bastien@ido-clarity.com](mailto:bastien@ido-clarity.com) to get other default parameters added.
It appends the Asana ticket ID to the end of the URL like so:
https://HIDDEN.m.pipedream.net/?taskGid=0123456789
This is exactly what I see in my Pipedream Logs.
I see the calls being made to my Workflow.
It seems like you have set up the Asana workflow to trigger an outgoing GET request to your Pipedream workflow. If you’re experiencing any issues or need help with the Pipedream workflow itself, please provide more information about the steps in your workflow or any specific issues you’re encountering. This will help me better understand your use case and provide assistance accordingly.
What are the steps for the workflow that is running out of memory?
Also: too bad we still can’t profile memory usage! 
I made a video for you.
Thanks for the video! 
One issue I’ve noticed with OOM errors is that Pipedream seems unable to show which steps have run or not.
So it could have failed on the very first or the very last steps, but Pipedream won’t show any successful steps either way.
That is unfortunately a limitation of the container runtime
And definitely for long workflows with lots of steps, this tends to happen much more often.
One way we’ve managed to mitigate this a bit is by splitting workflows in two (or more), connected with an HTTP call.
It costs more credits this way, but increasing the memory further costs more credits also anyway. :man-shrugging:
And the benefit is that you get a better idea of which half of the workflow is having the memory issue.