Can the Pipedream API be Used to Programmatically Create Sources and Workflows, and What are its Limitations?

This topic was automatically generated from Slack. You can find the original thread here.

Can you use the pipedream api to programmatically create sources and workflows in pipedream (i.e create 15 source/workflow pairs that execute the same logic just for different sources/accounts) - what are the limitations of the api?

Based on the information provided in the Pipedream documentation, the REST API currently supports a limited set of actions, such as retrieving workflow events. However, the ability to create, update, and delete workflows programmatically via the API is planned for the future, as mentioned in a GitHub issue. Therefore, at this time, you cannot use the Pipedream API to programmatically create sources and workflows. As for the limitations of the API, the documentation does not provide specific details on API rate limits or other restrictions. However, it is mentioned that event sources are subject to the same limits as Pipedream workflows, with the exception that sources have a default timeout of 5 minutes and a fixed memory allocation of 256MB. If you have a specific use case or need further assistance, you might want to follow the progress of the GitHub issue linked above or reach out to Pipedream directly for more information on their upcoming API features. If you need immediate support or have more questions, please visit https://pipedream.com/support.

would love to hear your specific use case so I can share with the team!

I have an GPT-based inbox assistant workflow that I’ve deployed for a few clients so far (I have it monitoring contact us inboxes and sales inboxes each with a custom ruleset for drafting/what actions to take for each scenario) and I am considering productizing it further but I want to validate demand first. I am launching a cold outreach campaign to heads of sales/cold email lead gen agencies to gauge interest and would like to be able to include a live demo (email this address to see how it works) and this would basically be just new email addresses in my domain + the workflow (and same GCP project) which I have figured out how to programmatically customize to a use case. If there is demand, I would likely keep hosting the mvp on pipedream at first before building a hosted app since it would allow me to monetize more immediately. Are there plans to add programmatic workflow creation?

that’s awesome thanks. Yes, we plan to support programmatic workflow creation.

Are you applying different rules to all your customers or would you ever want to apply the same logic / workflow to all users? i.e. if you had the option to dynamically reference user credentials, running a single workflow for all your customers, would that help? Just curious what shared logic there is and how you’d want this to work in an ideal case.

That’s great! I’d love to use that feature when it releases.

And it depends but more towards the second - that would help a lot. The workflow is fairly simple and reusable as the primary difference between each deployment is the drafting and action ruleset, which are actually just natural language prompts for GPT-4 function calls, and pretty much all hard-coded logic is encapsulated in a few classes I call from the handler. This has made it easy to scale, and take one improvement and deploy it for another client copy paste. Ideally, I would probably have a few workflows that would have different logic that would be tailored for a specific use cause/tier and then each would reference user-specific variables to pass in for each execution.

Also happy to share the workflow if you’re curious, I’ve been meaning to open source a version of it I just haven’t gotten around to it yet

makes a lot of sense, thanks. I’ll share this feedback