Advice for a workflow that exceeds 5 minutes

This topic was automatically generated from Slack. You can find the original thread here.

Adam Gaskins : On the pipedream limits page it says: “If you need to run a workflow that exceeds 5 minutes, please reach out to our team.” Is this the correct avenue for that?

Dylan Sather (Pipedream) : yes, thanks for reaching out. Currently there’s not a direct way for us to increase this limit past 5 minutes (this is the feature request where we’re tracking raising this limit), but I’d love to hear your specific use case, if you don’t mind sharing, and I may be able to help suggest a workaround.

Adam Gaskins : Cool! We have one system that can call a webhook and pass a CSV of image URLs as output, and another system that needs to receive the images as actual files in an SFTP server, so it seemed like a perfect opportunity to use pipedream to glue those together. However, there’s a lot of images and it would take much longer than 5 minutes even when downloading multiple images to the SFTP concurrently

Dylan Sather (Pipedream) : Great thanks. I’m developing an example for you on how I’d approach this and I’ll share it when I’m done

Dylan Sather (Pipedream) : do you have docs on the webhook? I’d like to see specifically how the CSV arrives in the HTTP request

Dylan Sather (Pipedream) : check out this GitHub repo, specifically the emit-rows-of-csv-as-distinct-events.js file and the README for instructions on how to deploy this. This is a custom event source I created. Anyone can author their own event source, if you’ve ever got a custom use case like this.

This source receives a CSV in the body of the HTTP request (it assumes the data is sent using a Content-Type of text/csv and assumes the CSV is the only thing in the HTTP payload). It parses the CSV and emits each row as its own event. This allows you to trigger a Pipedream workflow on each row, so that you’re only uploading a file at a time. Just visit the source in the UI (https://pipedream.com/sources) and click the Create Workflow button, which will create a workflow that runs on these emitted events.

This is the “Pipedream way” to handle this: if you’re receiving a batch of records, it’s often easiest to separate the records into units (like rows) and process them independently. That generally improves the readability of the workflow and lets you take advantage of features like concurrency and throttling that are helpful when you’re processing a lot of events against a target API.

Let me know if that makes sense!

Dylan Sather (Pipedream) : btw here’s a small example. If you send this CSV:

name,title
Luke,Jedi
Leia,General

to this event source, it’ll emit two structured JavaScript objects - one for each row of data:

Adam Gaskins : Ahh, that’s super helpful!

Adam Gaskins : I think that should work perfectly

Dylan Sather (Pipedream) : Great!

Adam Gaskins : I’ll work on this tomorrow and let you know, but thanks so much for the help!