How to Avoid Out of Memory Issues during Workflow Execution with Large Purchase Orders?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Hi ,
We’re preparing to take our workflow live and want to make sure we’re set up to avoid Out Of Memories issues at scale. The workflow has ~9 steps and processes purchase orders that can have hundreds of line items.
Questions:
Does each step’s $return_value stay in memory for the entire workflow execution, or is there any garbage collection between steps?
Would consolidating pure transformation steps into fewer steps meaningfully reduce memory usage, or is the overhead mostly in our data, not the step count?

  1. Is there a recommended way to monitor per-step memory consumption? We’re considering logging process.memoryUsage().rss at the start/end of each step.
  2. For workflows that process variable-size payloads (10 items vs 500 items), are there any best practices or memory limits we should be aware of?
    Any guidance would be really helpful. Thanks!
    FYI @U0AD20REN4T,
2

Hi, for your first question, steps’ return values are available for any step within the workflow to use, so there is no ‘garbage collection’.

As far as I know there is no recommended way to monitor per-step memory usage.

You probably already know this but you can increase the memory available for a workflow, which also makes it cost more credits per execution since it’s using more resources.

I would recommend doing all of your data transformation in as few steps as possible.

Our docs also mention other limits you may find useful to know: Limits - Pipedream