How to call a workflow from another?

[ericwu91]

I have a workflow whose code will (or should) be reused across multiple workflows.
I’ll provide a quasi-fictitious example:
Workflow “GET” retrieves data from a datastore by a given key, process it (adds extra properties) and returns it.
Workflow “PUT” receives a key, attempts to find it in the same datastore, retrieves it’s data, process it, and updates it incrementing a property.
Workflow “PATCH” receives a key, attempts to find it in the same datastore, retrieves it’s data, process it, updates it with extra parameters received in the body.

As you can see, both PUT and PATCH uses the same data GET returns, but this code is (tri)plicated.

Resources I found so far (and why they don’t solve my problem):

1. awaiting a $.send.http() - As per the docs, you cannot access the response since they are resolved after the workflow is finished.
2. axios - If, for instance, PATCH calls GET, I’ll have the latter’s computational time counted twice: Once for GET itself, and once for the time PATCH waits for GET to finish.
3. Custom Actions - Creating a custom action would work, but it can’t access a datastore.
4. Custom Events - I would need the workflow to trigger the event and get its data, not be triggered by it.
5. $.flow.suspend() - (as suggested by David Treves on slack) This would work as a workaround. I can pass resume_url as a parameter to GET, althought resuming the workflow would incur in an additional invocation cost.

Is there an official way to accomplish this?

[pierce]

Hi,

Just want to clarify you can use a Data Store within a custom action, this might be your most flexible option to reduce code duplication.

The link you’re referring to is the $.service.db which you’re right, is only available in sources. It’s similar wording, but a Data Store is actually a different concept in Pipedream.

To add a Data Store to a custom action, just include it as a prop:

export default {
  name: "Sample Datastore",
  version: "0.0.1",
  key: "sample-datastore",
  description: "",
  props: {
    db: {
      type: "data_store",
      label: "Datastore",
    },
  },
  type: "action",
  methods: {},
  async run({ $ }) {
    const value = await this.db.get('some_key_here');
    $.export("data", value);
  },
};

A Data Store is just another prop. More examples for use in both custom actions and Node.js code steps here:

Using Data Stores

Just a heads up, you’ll still have to configure which Data Store belongs in that implementation of a custom action in your workflow.

I hope this helps!

[ericwu91]

Hi, Pierce.
Thanks for the info. Nice to know that we can, in fact, create actions to access data stores.

I’d like to point out, though, that creating a custom action for establishing communications between workflows feels clunky. I feel like this is the kind of action that would bring workflows to yet another level. Shouldn’t there be an officially-supported action/step for that?

Are there plans in implementing this?

[pierce]

Hi @ericwu91

Custom actions are actually going to be manageable from your workflows, without installing and using the PD CLI tool to publish them.

That should help with the ease of use, so you can easily code and publish them without leaving the Pipedream dashboard.

This feature is in the works, and we’ll be making a big announcement when this is available.

If you need the results of Workflow A to be available in Workflows B & C, currently the only option is to use an HTTP request to trigger and await the response from Workflow A in Workflows B, C.

If you don’t need the response from Workflow A, then I recommend checking out the Subscriptions API: REST API

You can have one workflow subscribe to others with this interface. We also have built-in actions to interact with Subscriptions. This short video goes over an example of subscribing to all workflows:

Log workflow errors to AWS CloudWatch

Stay tuned for an announcement over the custom action management updates, I really think that will help with the DX clunkiness you’re referring to with that option.

[ericwu91]

Awesome!

I checked out the Subscriptions API before, but I thought it would be overkill.
For now, I’ll use the $.flow alternative.

Will be waiting for the improvements on Actions; Having to use the CLI breaks a little of the experience of having the codebase entirely browser-editable.

Thanks for the answers!

[ericwu91]

I had an insight still relating to this topic.
What if we could convert an existing step into a custom action? Something like a context action while creating a step inside a workflow.

Is that a valid option?

In my understanding, if a step were to be created with a modular, reusable point of view, that would allow for reducing duplicated code.

[pierce]

@ericwu91 we are actually in the middle of building exactly that

You can head over to the Alpha settings on your account and enable it.

You will need to define a key, name, version and type properties on the action, and after testing the Node.js code step it will unlock the ability to publish it to your account:

Here’s an example with all minimum fields needed to convert a Node.js code step into a custom action:

// To use any npm package on Pipedream, just import it
import axios from "axios"

export default defineComponent({
  key: 'pierce-pokemon-example',
  name: 'Get Charizard',
  version: '0.0.1',
  type: 'action',
  async run({ steps, $ }) {
    const { data } = await axios({
      method: "GET",
      url: "https://pokeapi.co/api/v2/pokemon/charizard",
    })
    return data.species
  },
})

After you publish a custom action from a Node.js step, you can access it within your Components area for easy reference:

To use this published action, open the My Actions section in a new workflow step.

[ericwu91]

You guys rock.

Will keep an eye for it.

[ericwu91]

@pierce , tested the new Publish Custom Action, and loved it.
I have a few doubts about it that might not relate to this main topic. I’ll post them below, but if you think we should open a new topic, I’ll be happy to move them.

Also, apologies in advance if these questions are being tackled by the team, the feature being in alpha, and all. I get too excited by brand new, shiny things

• Right now, the Components page points to the custom action published from within a step, along with it’s name, description and version provided in the code.
  However, the only way to edit it now is by entering the workflow from which it was created, and edit the code (and version) from there. I assume this will be resolved with the new Custom Action GUI?
• Related to the previous: If the action was created from a step, shouldn’t the original (step) be replaced by the new action - instead of having the code being shown? (Again, I believe this was kept this way because there wouldn’t be any way of editing this action otherwise).