How Can I Publish Python Steps Developed Locally and Integrate with Pipedream Using Github Actions?

Help
1

This topic was automatically generated from Slack. You can find the original thread here.

hello all, really happy to have found this slack workspace.
we’re trying to integrate pipedream deeply into our stack, and I have a few basic questions for starters. It would be amazing if someone could help me out.
Im trying to understand how to:
• publish Python steps that I develop locally on my machine.
• integrate the pd tool to publish these python steps that we version with Git via github actions

2

all the docs about writing custom components are using nodejs/typescript

3

to supply API credentials to the pd tool when running it inside github-actions, is my only option to edit the config file and insert the API key there that I pass to a github-actions step via github-actions secrets?

4

Hi there, welcome! Right now our components only support Node and TypeScript, but good to hear you’re interested in extending support for Python.

5

FYI we’re actively working on a git integration, which will let you serialize workflows as code and link projects within Pipedream to a git repo, so you can develop locally — it sounds like this might be a good workaround for you, when we start rolling it out in the coming weeks and months

6

that sounds great :slightly_smiling_face: love that feature.
In the meantime I wanted to be able to push Python code using the pd CLI tool from a local FS, to a component, or via any method for that matter.
I can then script it myself from github-actions to push code to pipedream on push events.
Can pd push Python code to pipedream?

7

Yea, the git integration I think is what you’re looking for and should make a lot of the local development easier. In the meantime, if you want to publish Python code locally so you can use it in your workflows, publishing a PyPi package to use in a workflow is probably the best option. FYI the package has to be public (we don’t have a way to pull from private registries).

8

Is that feasible?

9

thats tricky :disappointed: we cant really publish our internal code as public PyPi packages.
being able to push Py code to pipedream is really important for us.
In particular we have some rather complex dataprocessing pipelines, where the logic is spread across multiple files; so pasting those via the step/component UI editor is not ideal.
thanks for giving me a straighforward answer. I spent quite a bit of time in the docs and forums looking for an answer to this

10

Ah, gotcha, okay thanks for the info. cc @U02SX7ETBFB might have some other ideas in the meantime

11

Hello,
I am looking to publish my python code into github, what is the best way to do that currently.

Thanks
Brice

12

+1 for python support. We want to publish a couple custom action and our main language is python. It will be very useful for us to have Python support.