This topic was automatically generated from Slack. You can find the original thread here.
Hi! I’d like to create a workflow that scrolls a huge MongoDB index and write some updates on SendGrid. The full process requires ~1:30h and it is a very basic ETL job. What are the best practices for doing something like that with Pipedream? Any example to point out?
Hello , I don’t think running long process is possible in Pipedream. Maybe you can off-load the long running process on a VM/ETL framework on your Cloud provider, then have it call Pipedream webhook when things finished.
For more context, Pipedream uses AWS Lambda to execute actions, hence, Pipedream is limited by AWS Lambda limitation of 15 minutes execution time.