This topic was automatically generated from Slack. You can find the original thread here.
Hi I am wondering how I should manage the memory in my workflows : Memory should be fixed depending of the quantity of data processed in my bigger step within my workflow, or is it related to quantity of data & process for the entire workflow ?
Memory management in Pipedream workflows is related to the entire workflow, not just the largest step. By default, workflows run with a certain amount of memory, which you can increase up to a maximum limit. This memory setting affects the whole workflow and all the steps within it. If you’re processing a lot of data or need more compute power, you can increase the workflow’s memory, which also proportionally increases the CPU available to your workflow. However, keep in mind that Pipedream charges credits proportional to your memory configuration. If you increase the memory, you will be charged more credits for each execution. For more detailed information on how memory settings affect credits and performance, you can visit the following Pipedream documentation pages: - Workflow Memory Settings - Pricing and Credits - Workflow Limits
Hi ! To consume less memory, is it better to have one single step with my whole JS logic, or is it better to have smaller step, each one with a specific JS logic ?
Based on the Pipedream documentation and best practices, it’s generally better to organize your code into smaller, more focused steps. This can make your workflows easier to understand and maintain. Each step in a Pipedream workflow is a separate Node.js process, and memory is allocated to each step independently. Therefore, splitting your logic into multiple steps should not consume more memory than a single step containing all the logic. However, if you have specific performance concerns or if your logic is highly interdependent, it might make sense to keep related logic together in a single step. It’s important to balance readability and maintainability with performance. If you have further questions or need assistance with optimizing your workflow, please feel free to reach out at Pipedream Support.