Is There a Better Way to Implement Looping Over Outputs of a Step in Pipedream?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

Looping over outputs of a step.

So there a lot of smart peeps in here.

Currently when i have an array of objects i want to loop over i like to track the progress of each workflow executed on the individual object.

My current set up is to have

workflow 1 that builds the array of objects and calls work flow 2 to process each object
workflow 2 that processes each object sent to it via the webhook call from workflow 1.

Is there a better way to implement this in pipedream?

I set it up this way so it kind of operates like celery with a Producer/Broker and consumer

2

For now yes that’s how we recommend tackling this type of problem.

But in the future we’ll be releasing a native loop/iteration action to process your records individually in a sub-workflow to overcome the 12.5 minute timeout restriction even with large amounts of records on an async operation.

3

Awesome! Super excited for that

4

I’m excited too :slightly_smiling_face: it’s going to be awesome