How Can I Decrease the Memory Requirements of a Complex Automation Workflow to Prevent Running Out of Memory?

Help
21

Where do I add this code? On the “Custom_request” step in the first workflow, I put the HTTP request configuration to “Post” and pointing it at the URL created in the first step of the second workflow.

Do I add variables from the first workflow as parameters that I’m pushing in to the second workflow? Or somewhere else?

22

You should add the code in the first workflow using the “Send any HTTP Request” action from the HTTP / Webhook app. Here’s how to do it: 1. In the first workflow, add a new step and search for the HTTP / Webhook app. 2. Select the “Send any HTTP Request” action. 3. Configure the action with the following settings: - Method: POST - URL: The URL of the HTTP trigger in your second workflow - Headers: Add a header with the key “Content-Type” and value “application/json” - Body: Add a JSON object containing the variables you want to pass from the first workflow to the second one. For example: { "row": rowVariable, "originalOutput": originalGPTOutput, "recentOutput": recentGPTOutput } In the second workflow, you can access these variables using steps.trigger.event.body. For example: ```
const row = steps.trigger.event.body.row; const originalOutput = steps.trigger.event.body.originalOutput; const recentOutput = steps.trigger.event.body.recentOutput;

23

is it possible to just pass the variables as parameters or any other way where I can just select them from my list?

24

Yes, you can pass the variables as query parameters in the URL when making the HTTP request from the first workflow to the second workflow. Here’s how to do it: 1. In the first workflow, add a “Send any HTTP Request” action from the HTTP / Webhook app. 2. Configure the action with the following settings: - Method: POST - URL: The URL of the HTTP trigger in your second workflow, with the variables appended as query parameters. For example: https://your-second-workflow-url.pipedream.net?row=rowVariable&originalOutput=originalGPTOutput&recentOutput=recentGPTOutput Replace rowVariable, originalGPTOutput, and recentGPTOutput with the actual variable values from your first workflow. 3. In the second workflow, you can access these variables using steps.trigger.event.query: ```
const row = steps.trigger.event.query.row; const originalOutput = steps.trigger.event.query.originalOutput; const recentOutput = steps.trigger.event.query.recentOutput;

25

actually, the variables seem to be quite long. I don’t think passing them as parameters will work well.

Can you give me a more detailed instruction on adding the variables in the headers?

For example, you said to add a “Key” for “Content Type” – but I have no option to add any “keys”, just “Header Name” and “Header Value”

26

I apologize for the confusion. In the “Send any HTTP Request” action, you should add the variables in the request body, not the headers. Here’s a step-by-step guide: 1. In the first workflow, add a new step and search for the HTTP / Webhook app. 2. Select the “Send any HTTP Request” action. 3. Configure the action with the following settings: - Method: POST - URL: The URL of the HTTP trigger in your second workflow 4. In the “Headers” section, add a new header with: - Header Name: Content-Type - Header Value: application/json 5. In the “Body” section, select “Raw” and choose “JSON” from the dropdown menu. 6. Add a JSON object containing the variables you want to pass from the first workflow to the second one. For example: on { "row": "rowVariable", "originalOutput": "originalGPTOutput", "recentOutput": "recentGPTOutput" } Replace rowVariable, originalGPTOutput, and recentGPTOutput with the actual variable values from your first workflow. In the second workflow, you can access these variables using steps.trigger.event.body: ```
const row = steps.trigger.event.body.row; const originalOutput = steps.trigger.event.body.originalOutput; const recentOutput = steps.trigger.event.body.recentOutput;

27

I tested this step, and all I got in the return_value is “success: true”

How do I check that the custom post request is actually adding the JSON variables?

28

To verify that the custom POST request is sending the JSON variables correctly, you can add a temporary step in the second workflow to log the received variables. In the second workflow, add a new “Run Node.js code” step right after the HTTP trigger step, and use the following code: ```
console.log(“Received row:”, steps.trigger.event.body.row); console.log(“Received originalOutput:”, steps.trigger.event.body.originalOutput); console.log(“Received recentOutput:”, steps.trigger.event.body.recentOutput);

29

I have this error:

ReferenceError: steps is not defined
    at Object.<anonymous> (/tmp/__pdg__/dist/code/cf4bf7e54dafda4f2baad58106b8ad5afeef15915bc823c618ff626c96289a82/code.js:1:30)
    at Module._compile (node:internal/modules/cjs/loader:1256:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1310:10)
    at Module.load (node:internal/modules/cjs/loader:1119:32)
    at Module._load (node:internal/modules/cjs/loader:960:12)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/translators:169:29)
    at ModuleJob.run (node:internal/modules/esm/module_job:194:25)
30

I apologize for the confusion earlier. It seems that the error is related to the steps variable not being defined. Please make sure that you’re using the correct syntax for the run method in your code. It should look like this: export default defineComponent({ async run({ steps, $ }) { // Your code here }, }); Make sure you have the curly braces { steps, $ } in the run method’s parameters. This will properly define the steps variable and should resolve the error.

31

here is the code I have:

export default defineComponent({
async run({ steps, $ }) {
console.log(“Received row:”, steps.trigger.event.body.row);
console.log(“Received originalOutput:”, steps.trigger.event.body.originalOutput);
console.log(“Received recentOutput:”, steps.trigger.event.body.recentOutput);
}

Here is the error:

SyntaxError: Unexpected token (6:1)
    at pp$4.raise (/var/task/node_modules/acorn/dist/acorn.js:2927:15)
    at pp.unexpected (/var/task/node_modules/acorn/dist/acorn.js:698:10)
    at pp.expect (/var/task/node_modules/acorn/dist/acorn.js:692:28)
    at pp$3.parseObj (/var/task/node_modules/acorn/dist/acorn.js:2563:14)
    at pp$3.parseExprAtom (/var/task/node_modules/acorn/dist/acorn.js:2302:19)
    at anonymous.parseExprAtom (/var/task/node_modules/acorn-node/lib/dynamic-import/index.js:77:117)
    at anonymous.parseExprAtom (/var/task/node_modules/acorn-node/lib/import-meta/index.js:27:102)
    at pp$3.parseExprSubscripts (/var/task/node_modules/acorn/dist/acorn.js:2129:21)
    at pp$3.parseMaybeUnary (/var/task/node_modules/acorn/dist/acorn.js:2106:19)
    at anonymous.parseMaybeUnary (/var/task/node_modules/acorn-node/lib/private-class-elements/index.js:122:54)
32

I apologize for the confusion earlier. It seems that the error is related to the formatting of your code. Please make sure that you’re using the correct syntax for the defineComponent function. Here’s the corrected version of your code: ```
export default defineComponent({ async run({ steps, $ }) { console.log(“Received row:”, steps.trigger.event.body.row); console.log(“Received originalOutput:”, steps.trigger.event.body.originalOutput); console.log(“Received recentOutput:”, steps.trigger.event.body.recentOutput); }, });

33

now i am receiving another error:

 at Object.run (file:///tmp/__pdg__/dist/code/9dae93210af015c7cc6d84bd18d51503696e0377d3559a50041205e6134e4c7e/component.mjs:3:59)
    at null.executeComponent (/var/task/launch_worker.js:229:53)
    at MessagePort.messageHandler (/var/task/launch_worker.js:726:28)
34

I apologize for the inconvenience. To better assist you, could you please share the complete code you’re using in your Pipedream component? This will help me identify the issue and provide a solution.

35

In the custom_request step, I have the following setup:

• Header:
◦ Header Name: Content-Type
◦ Header Value: application/json
• Body
◦ content-type: application/json
◦ Keys: row, originalOutput, recentOutput
◦ Values: Variables defined by selecting path in the workflow
In the Node step, I have the following code:

export default defineComponent({
async run({ steps, $ }) {
console.log(“Received row:”, steps.trigger.event.body.row);
console.log(“Received originalOutput:”, steps.trigger.event.body.originalOutput);
console.log(“Received recentOutput:”, steps.trigger.event.body.recentOutput);
},
});

36

I apologize for the confusion earlier. Based on the error message you provided, it seems that the issue might not be directly related to your code. To further investigate the issue, please try the following steps: 1. Make sure there are no extra characters or incorrect formatting in your code. 2. Remove any unnecessary spaces or line breaks. 3. Save your workflow and try running it again. If the issue persists, please provide more details about your workflow, such as the trigger type and any other steps involved. This will help me better understand the context and assist you in resolving the issue.

37

I am not running the entire workflow (because it takes roughly 10 minutes to run in its entirety). I am simply testing the individual steps. Could this be the issue?

38

Testing individual steps should not cause any issues, as long as the required data is available for each step. However, if a step relies on data from a previous step, you might encounter errors when testing the steps individually. In this case, you should ensure that the required data is available for the step you’re testing, either by providing sample data or by running the steps that generate the required data before testing the step in question.

39

okay – let’s forget about this method. How do I pull in the required data into the variables from the HTTP post into the second workflow? I’ll test it this way

40

To receive data from an HTTP POST request in your second workflow, you can use the HTTP trigger. Here’s how to set it up: 1. Create a new workflow. 2. Select the “HTTP” trigger. 3. Copy the endpoint URL provided by the HTTP trigger. You’ll use this URL to send the POST request from your first workflow. In your first workflow, use the “Send POST Request” action from the HTTP / Webhook app to send the data to the second workflow: 1. Search for the HTTP / Webhook app from the step menu. 2. Select the “Send POST Request” action. 3. Paste the endpoint URL from the second workflow’s HTTP trigger. 4. Configure the headers and body as needed. Now, when you send the POST request from the first workflow, the second workflow will be triggered, and you can access the data from the request using steps.trigger.event.body. For example, in a Node.js code step in the second workflow: ```
export default defineComponent({ async run({ steps, $ }) { console.log(“Received data:”, steps.trigger.event.body); }, });