Developing code steps in a workflow locally?

New to Pipedream, coming from Cloud Function-land.

Is there anyway to take a code step in an existing workflow, and develop it locally, and push changes back to pipedream?

The code editor provided in the browser is not bad, but I’d prefer to write code in my IDE, with my own version control, tests, etc… (and maybe burn a few less invocations while I’m working through stuff… hehe…)

The CLI seems interesting, but as far as I can tell, you can only deploy single components, and not work within full workflows.

It if matters, most of my workflows are just Webhook => Code Step. I’m looking for a better workflow than writing code in the browser editor.

FWIW, answering my own question:

https://pipedream.com/docs/cli/reference/#command-reference

to start:

pd dev myFile.js

to edit:

pd dev --dc {{job_id}} myFile.js

workflow is really nice… here’s a template i used to get started, just in case it helps anyone else


export default {
    name: "my source name",
    version: "1",
    description: "my source description",
    props: {
        http: {
            type: "$.interface.http",
            customResponse: true
        }
    },

    //methods written outside of run get hooked in here!
    methods: { bar },
    
    //lifecycle hooks if needed; called by pipedream when your source changes
    hooks: {
        async activate() {},
        async deactivate() {},
        async deploy() {},
    },
    
    //this runs inside pipedream with every invocation    
    async run(event) {
     
        const foo = this.bar
        
        //respond to invocation
        this.http.respond({
            status: 200,
            body: foo
        });

        //emit data for workflow
        this.$emit(foo);
    },
};

//methods can go here
function bar() {
    return `foo!`
}

function test(){
    return
}

//you can also run stuff outside of pipedream locally for testing, etc...
test(bar)

@ak Yes, Pipedream components are a great way to run Node code in response to webhooks / timers / other app events.

Re: your original question, we’re actually working on an API / CLI for workflows right now. Initially, that will let you serialize a workflow (both the code and configuration) to a text-based format that you can work with locally and commit to Git, but tests will likely happen in the UI (we are still working on these details).

In the future, we’re planning to let you test workflows locally so that you don’t incur invocations on Pipedream infra.

Let me know if that helps or if you have any other feedback / questions. I really appreciate you contributing the pd dev example back to the topic — that really helps the community.

Thanks,
Dylan

1 Like

totally. i’m (candidly) surprised at how much better pipedream is than some of the other SaaS automations tools while being MUCH lighter than lamdas in the cloud. It’s a really nice balance.

if you’re curious, here’s my local workflow… basically one file (LOVE THAT!) with 4 shells open:

  • pd dev (watch for changes)
  • pd logs (so i don’t need the web browser)
  • my jest tests on my code
  • a scratchpad for doing random stuff

as a feature request, it would be cool if you could see the data emitted (this.$emit()) by the source in pd logs

1 Like

@ak that’s awesome! Really cool to see that setup.

Re: seeing emitted events, have you used the pd events command? That should stream emitted events to standard out, as events are emitted. Or have you seen that command, and are you looking to see all logs / events in pd logs?

funny you mention it… i just discovered pd events -f and it’s way better… thanks!

1 Like