Is There an Alternative to the Long Execution Times Caused by Python Runtime in Pipedream Loading Imported Packages Upon Each Workflow Execution?

1

This topic was automatically generated from Slack. You can find the original thread here.

Hi everyone! :wave: It seems that the Python runtime in Pipedream loads the imported packages upon each execution of the workflow. This contributes to very long execution times, especially if the library is large or complex. Is there any alternative?

2

Hi , it would be great if you could create a GitHub issue and share the steps to reproduce your issue. Then Pipedream team can try to investigate and improve the python code step.

For now, I couldn’t think of any alternative without detailed information. As Pipedream is serverless by nature, you can search the Internet for best practices for implementing serverless functions.

3

I was wondering if there is a way to keep the workflow “warm”, so that it doesn’t need to load packages every time it’s invoked. Instead of having cold starts every time.

4

I was wondering if there is a way to keep the workflow “warm”, so that it doesn’t need to load packages every time it’s invoked
there’s a feature to keep to workflow “warm”, but it’s to eliminate the cold start, not to improve the performance. I think currently there’s no way to keep it warm like you described

5

There are two types of cold starts: compute initialization & static initialization.

It sounds like Pipedream solves one, but not the other?

6

I wrote an answer about this on StackOverflow.