Offer pd dev for action development OR allow sources to call other (pipedream) sources

Feature Requests
1

I noticed that you can’t use pd dev when you’re authoring an action (only sources are supported).

If you’re writing a custom action, you have to pd publish each time you make a change AND increment the version number AND go to the UI and click the “update” button. That’s a lot of things just to check some code!

This is kind of a frustrating workflow, especially because (at least for me) writing code is often quite iterative where I am frequently adding a feature and then running it manually to see if it works.

Given that you can easily iterate on a custom source using pd dev, I figured I’d make two custom sources and have them communicate with each other, using a combination of pd dev --dc and pd events -f on both sources simultaneously. I realize this would count as double invocations for billing, but I’d take the double hit for the workflow improvement.

I then noticed that if component-Source-A tries to make a POST requests to component-Source-B, pipedream responds with a 400 … If I had to guess, this is probably a security feature, but it ostensibly blocks my workflow:

async run(event) {
    await axios.post(`https://someSource.m.pipedream.net`, someData) //always returns 400
}

So I guess this is a feature request to make it easier to iterate on custom actions locally OR let sources call sources!

But Why?

My use case is actually simple; I’m building a “serverless” slackbot that needs to:

  • chat: post messages in slack that maybe contain interactive buttons/forms
  • acknowledge: respond immediately whenever a form from a message is submitted
  • do-stuff: preform some async task… let chat know when it’s done

Given that there are 3 separate core services, i’d ideally like the business logic to live in 3 separate files, which i can maintain independently.

Originally the goal was to have chat be its own source and create a workflow for acknowledge => do stuff … but this didn’t work because I couldn’t quickly iterate on do stuff

Then I tried having chat, acknowledge, and do stuff be their own sources and make acknowledge POST to do stuff (which is where I encountered the 400).

I’m after a development workflow that lets me work on those 3 services in isolation. Doesn’t exist today (as far as I can tell).

So now I’m basically stuck with 2 sources: chat and acknowledge-and-then-do-stuff and I’m not thrilled with how tightly coupled the later source is…

My Ultimate Goal
In a perfect world, I can grant authorship to others who can edit the do stuff code without messing around with the chat or acknowledge code. chat and acknowledge should change very infrequently; these are my top level APIs. do stuff will change frequently and i’d actually like it to live in a github repo with ci/cd.

(don’t take this as negative feedback; pipedream is really cool and I’m sharing this use case to help the team think about dev workflow!)

2

@ak This is amazing feedback as always. I hear you completely and we plan to add pd dev support for actions. You can follow this issue to track that, and I let the team know you asked about it.

More generally, we’re working on workflow serialization / editing workflows locally, and we’re targeting a development environment that would afford the general development pattern you mention. I agree it would be great to support quick iteration / hot reload for all Pipedream “atoms” (sources, actions, workflows, etc.)

Re: sources calling other sources, that should be possible. A 400 error returned from a source indicates an error in the target source. Can you check the Logs for that source to see if you observe anything there?

In addition to making an HTTP request to a source, you can call it from another source by adding source #2 as a listener. Take a look at these docs for specific information on how to do that. Once you’ve connected source #2 as a listener for source #1, any calls to this.$emit(event) from source #1 will trigger source #2, passing event in the run method.

Let me know if that works for you for now.

1 Like