How to Handle Long-Running Deep Research Tasks Using Paid Features and Memory Docs for Existing Integrations?

This topic was automatically generated from Slack. You can find the original thread here.

Heya team, how are people handling long-running deep research tasks? I signed up for paid becuase I thought it would give me that power, curious if anyone has tips? I don’t really have time to trial and error here:

I read the troubleshooting memory docs, but im unclear how I should be using it for using the existing integrations? Anyone have a fix? thanks in advance

What type of deep research tasks are you doing? Workflow execution has a hard cap of 12.5 minutes, but you can suspend execution basically indefinitely, and resume later via a callback or by polling for updates to an async job, etc.

Just like “do a bunch of web searches on XYZ and dump me back output”

Could you please walk me through how you’d do that in pipedream?

I am considering switching to n8n, no pressure :sweat_smile:

Is this something you want to run on an ongoing basis? Or as a one-off?

Looks like you want to run it every day?

I’m just prototyping here, but ideally i’d like the concept of an agent – a model that backs a long running task, and can call tools, wait and continue as needed. without worrying about timeout

That error that you’re showing is saying that the workflow execution ran out of memory, not time. So it’s possible the results it got back from one of the steps requires more memory in the execution environment, I’d try increasing the memory (and probably the timeout).

I will use this in my consulting work for clients so pretty high volume

But like isn’t the memory limit (256 MB) plenty large for an LLM outpiut?

No

That’s the amount of memory for the entire workflow execution, not strictly the data that’s returned from the LLM

Is there large overhead?

I can play with it :+1:

just didnt want to upsell at first challenge hit lol

Cool, and check this section of the docs out, it has some good info: GitHub

I believe timeout and memory settings can be configured on the free plan, but you’ll likely run out of credits very quickly

Yeah got it, thanks!