This topic was automatically generated from Slack. You can find the original thread here.
Is there any way to install custom libraries on pipedream? I’d like to use functions from the libraries I have created without having to include all the code in the workflow.
from custom_library import custom_function
Thanks for reaching out. Are you trying to include Node.js or Python packages?
Python
are you able to publish these as public PyPI packages? Since you can install any public package from PyPI, that’s one option if that works for you
That’s not really an options unfortunately
got it. What’s your ideal interface for loading custom_library into Pipedream? e.g. if you could write code locally, bundle your package with the rest of your workflow code, and deploy the workflow with all the code, would that work?
I’m not sure I totally understand. I’ve got an app folder that contains all the various packages I’ve written for my project. I’ve then got several scripts that I’d like to run sequentially using Pipedream. In each of these scripts, the functions are imported at the start of each script.
When you say deploy the workflow with all the code, do you mean to add all my library code into one mega code which I then keep at the top if each sequential run on Pipedream? If so, then whilst that would work, the readability would be quite poor
thanks, no I’m trying to understand how you share this code with other projects right now. PyPI packages are one way to distribute code for use in other applications, so when you say you want to run:
from custom_library import custom_function
if you don’t run that on PyPI, you’d either need to have a custom_library dir with the package code stored alongside the Pipedream workflow code, or e.g. we could provide a way for you to publish packages to Pipedream directly and have you reference shared libraries across workflows. So I’m just trying to understand the use case so we can ideate.
Is publishing to PyPI not an option because the code can’t be public? If you could publish a private package and then use it in Pipedream, would that work?
Publishing to Pipedream would definitely be an option. You’re correct, the code is proprietary so we can’t make it public.
At the moment we have a custom library dir with the packages.
Thanks. We’ll be shipping a Git integration soon with better ways to work with code locally. At that point, it should be easier to reference other local modules (e.g. you can keep them all in a single Git repo, and reference the custom functions in workflows). For now, you’ll need to include any custom functions directly in Python code steps.
Ok cool. Thanks Dylan! Any rough estimate for when you’ll have the git integration available?
in beta with our first customers, should have it out to this community within a few weeks for public beta! We announce early features like this in the announcements channel to this community / via our email mailing lists