This topic was automatically generated from Slack. You can find the original thread here.
what are some ways that I can decrease the memory requirements of a workflow? I keep running out of memory, and the whole automation stops.
Technical details:
• Trigger on new row in Google Sheets
• 106 steps
• First step is Node code to pull row number from trigger row
• Sequential calls to ChatGPT, Add Cell, ChatGPT, Add Cell, etc.
• I am adding 85~ rows at once
• Auto-retry is on
• 2048mb memory
• Throttle
◦ Limit: 1
◦ Every 5 seconds
• Concurrency
◦ 1 worker
◦ 10,000 queue size
Any suggestions on reducing memory load?
It seems like you have a complex workflow with many steps. To reduce memory usage, you can try the following: 1. Optimize your Node.js code: Review your custom code steps to ensure they’re efficient and not causing memory leaks. 2. Break the workflow into smaller workflows: Instead of having a single workflow with 106 steps, you can create multiple smaller workflows and use Pipedream’s HTTP actions to trigger them as needed. 3. Limit the number of rows processed at once: Instead of adding 85 rows at once, consider processing a smaller number of rows in each run and use the Scheduler trigger to run the workflow more frequently. Remember to monitor your workflow’s memory usage and adjust these suggestions as needed. If you still face issues, please visit https://pipedream.com/support for more support options.
I have managed to fix the memory issue without breaking the workflow up, but it seems like all of my workflows are failing between step 50 - 60, timing out (750s).
The timeouts are always happening on a GPT chat step, and roughly in the same spot of the automation.
Given how consistently it times out between steps 50 and 60 – what could be causes?
• Too many consecutive calls to the GPT3.5 16K API?
• An issue without how pipedream makes these calls?
• Some sort of non-memory-related soft-limit on workflow size that is preventing the proper functioning of the steps past a certain point?
It’s great to hear that you’ve resolved the memory issue. Regarding the timeouts between steps 50 and 60, there could be a few possible causes: 1. Rate limits or latency issues with the GPT3.5 16K API: If the API has rate limits or is experiencing high latency, it might cause timeouts in your workflow. 2. Issues with Pipedream’s handling of the API calls: There could be a bug or issue with how Pipedream is making the API calls to GPT3.5 16K API. 3. Workflow execution time limit: Pipedream workflows have a time execution limit of 12.5 minutes (750 seconds). If your workflow consistently takes longer than this limit, it will time out. To further investigate the issue, you can try the following: - Check the GPT3.5 16K API’s documentation for any rate limits or known latency issues. - Monitor the execution time of each step in your workflow to identify any bottlenecks or slow steps. - If you suspect an issue with Pipedream’s handling of the API calls, please visit https://pipedream.com/support for more support options. Remember to keep an eye on your workflow’s execution time to ensure it stays within the allowed limit.
The 750s (12.5 minutes) time limit applies to the entire workflow execution, not individual steps. If the total execution time of your workflow exceeds this limit, it will time out. To troubleshoot the issue, you can: 1. Monitor the execution time of each step in your workflow to identify any bottlenecks or slow steps. 2. Check the logs for any errors or unexpected behavior in the steps leading up to the GPT3.5 16K API calls. 3. Add console.log statements in your custom code steps to gather more information about the state of your workflow during execution. If you still face issues or suspect a problem with Pipedream’s handling of the API calls, please visit https://pipedream.com/support for more support options.
can you please give me step-by-step instructions on how I would break the current workflow up into 3 separate ones?
The current workflow is triggered when I add a row to a Google Sheet. I turn the row into a referenceable variable with Node, and then I alternate calling GPT and adding its response into a cell in the originating row until the end of the workflow.
If I end the first workflow on step 40 (let’s say, adding a cell of the GPT output from step 39) – what’s the best way to trigger the next workflow to continue with the second set of steps?
To break your current workflow into three separate workflows, you can follow these steps: 1. First Workflow: - Keep the Google Sheets trigger that initiates the workflow when a new row is added. - Perform the steps to turn the row into a referenceable variable with Node, and alternate calling GPT and adding its response into the cell in the originating row until step 40. - At the end of this workflow, use a “Send HTTP Request” action to make an HTTP request (e.g., a POST request) to the endpoint URL of the second workflow (which you’ll create in the next step). You can pass any necessary data from the first workflow to the second workflow as JSON in the request body. 2. Second Workflow: - Create a new workflow with an HTTP trigger. Copy the endpoint URL provided by the HTTP trigger. - In this workflow, continue processing the data from where the first workflow left off. You can access the data passed from the first workflow using steps.trigger.event.body. - Perform the steps until you reach the desired point (e.g., step 80). - At the end of this workflow, use a “Send HTTP Request” action to make an HTTP request (e.g., a POST request) to the endpoint URL of the third workflow. Pass any necessary data from the second workflow to the third workflow as JSON in the request body. 3. Third Workflow: - Create a new workflow with an HTTP trigger. - In this workflow, continue processing the data from where the second workflow left off. You can access the data passed from the second workflow using steps.trigger.event.body. - Perform the remaining steps to complete your process. By chaining the workflows using HTTP triggers and requests, you can break your large workflow into smaller, more manageable parts.
is there any way for me to mass delete steps in the workflow? Or do I have to duplicate my workflow 3x and then manually delete each step to break it into 3 parts?
Currently, there isn’t a built-in feature to mass delete steps in a workflow. You’ll need to duplicate your workflow 3 times and manually delete the steps in each duplicated workflow to break it into 3 parts. I understand this might be time-consuming, but it’s the current method to achieve your goal. If you have any suggestions or need further assistance, please visit https://pipedream.com/support for more support options.
For the HTTP request trigger – how do i pass information from the previous workflow to the second one?
For example, I need to pass 3 things from the first automation:
• The row variable (what row needs to be updated in the spreadsheet)
• The original output of GPT (an outline)
• The most recent output of GPT (for context)
All of these variables are stored in the first workflow.
To pass information from the first workflow to the second one using the HTTP request trigger, you can include the necessary data in the request body of the “Send HTTP Request” action in the first workflow. Here’s an example of how to do that: In the first workflow, at the end of the process (after step 40), add a “Send HTTP Request” action with the following configuration: { "method": "POST", "url": "https://YOUR_SECOND_WORKFLOW_URL", // Replace with the endpoint URL of your second workflow "headers": { "Content-Type": "application/json" }, "data": { "row": rowVariable, // Replace with the row variable from your first workflow "originalOutput": originalGPTOutput, // Replace with the original output of GPT from your first workflow "recentOutput": recentGPTOutput // Replace with the most recent output of GPT from your first workflow } } In the second workflow, create an HTTP trigger and use steps.trigger.event.body to access the data passed from the first workflow: ```
const row = steps.trigger.event.body.row; const originalOutput = steps.trigger.event.body.originalOutput; const recentOutput = steps.trigger.event.body.recentOutput;