Running workflows for your end users

Just like you can build and run internal workflows for your team, you can run workflows for your end users, too.

Whether you’re building well-defined integrations or more-autonomous AI agents, workflows provide a powerful set of tools for running code or pre-defined actions on behalf of your users. Pipedream’s UI makes it easy to build, test, and debug workflows.

What are workflows?

Workflows are sequences of steps triggered by an event, like an HTTP request, or new rows in a Google sheet.

You can use pre-built actions or custom Node.js, Python, Golang, or Bash code in workflows and connect to any of our 2,400 integrated apps.

Workflows also have built-in:

Read the quickstart to learn more.

Getting started

Create a workflow

Create a new workflow or open an existing one.

Add an HTTP trigger, configure OAuth

Add an HTTP trigger to your workflow.
Configure OAuth authorization on the trigger.

To securely run workflows for end users, configuring OAuth authorization on the trigger is optional but strongly recommended.

Create a Pipedream OAuth client to authenticate requests to the Pipedream API and workflows.

Configure accounts to use your end users’ auth

When you configure pre-built actions or custom code that connects to third-party APIs, you can link accounts in one of two ways:

Use your own account: If you’re connecting to an API that uses your own API key or developer account — for example, a workflow that connects to the OpenAI API or a PostgreSQL database — click the Connect account button to link your own, static account.

Use your end users’ auth: If you’re building a workflow that connects to your end users’ accounts — for example, a workflow that sends a message with your user’s Slack account — you can select the option to Use end user’s auth via Connect:

When you trigger the workflow, Pipedream will look up the corresponding account for the end user whose user ID you provide when invoking the workflow.

Connect a test account

To run an end-to-end test as an end user, you need to have users and connected accounts in your project. If you already have a development account linked, you can skip this step.

If you don’t, the fastest way to do this is on the Users tab in your Pipedream project:

You’ll see there’s a button to Connect account
Go through the flow and make sure to create the account in development mode
Note the external user ID of the account you just connected, you’ll need it in the next step

Generate a test request

Test events are critical for developing workflows effectively. Without a test event, you won’t be able to test your workflow end to end in the builder, see the shape of the event data that triggers the workflow, and the lookup to use your end user’s auth won’t work.

To generate a test event, click Send Test Event in the trigger, and fill in the event data. This will trigger the workflow and allow you to test the workflow end to end in the builder.

Make sure to include these headers in your test request:

x-pd-environment: development
x-pd-external-user-id: {your_external_user_id}

Deploy your workflow

When you’re done with the workflow, click Deploy at the top right.

Invoke the workflow

If you’re using TypeScript or a JavaScript runtime, install the Pipedream SDK. Pipedream also provides an HTTP API for invoking workflows (see example below).

npm i @pipedream/sdk

To invoke workflows, you’ll need:

The OAuth client ID and secret from your OAuth client in step 2 above (if configured)
Your Project ID
Your workflow’s HTTP endpoint URL
The external user ID of the user you’d like to run the workflow for
The Connect environment tied to the user’s account

Then invoke the workflow like so:

import { createBackendClient, HTTPAuthType } from "@pipedream/sdk/server";
 
// These secrets should be saved securely and passed to your environment
const pd = createBackendClient({
  environment: "development", // change to production if running for a test production account, or in production
  credentials: {
    clientId: "{oauth_client_id}",
    clientSecret: "{oauth_client_secret}",
  },
  projectId: "{your_project_id}"
});
 
await pd.invokeWorkflowForExternalUser(
  "{your_endpoint_url}", // pass the endpoint ID or full URL here
  "{your_external_user_id}" // The end user's ID in your system
  {
    method: "POST",
    body: {
      message: "Hello World"
    }
  },
  HTTPAuthType.OAuth // Will automatically send the Authorization header with a fresh token
)

import { createBackendClient } from "@pipedream/sdk/server";
 
// These secrets should be saved securely and passed to your environment
const client = createBackendClient({
  environment: "development", // change to production if running for a test production account, or in production
  credentials: {
    clientId: "{oauth_client_id}", 
    clientSecret: "{oauth_client_secret}"
  },
  projectId: "{your_project_id}"
});
 
const response = await client.invokeWorkflowForExternalUser(
  "{your_endpoint_url}", // pass the endpoint ID or full URL here
  "{your_external_user_id}" // The end user's ID in your system
  {
    method: "POST",
    body: {
      message: "Hello World"
    }
  }
)

# First, obtain an OAuth access token
curl -X POST https://api.pipedream.com/v1/oauth/token \
  -H "Content-Type: application/json" \
  -d '{
    "grant_type": "client_credentials",
    "client_id": "{oauth_client_id}",
    "client_secret": "{oauth_client_secret}"
  }'
 
# The response will include an access_token. Use it in the Authorization header below.
 
curl -X POST https://{your-endpoint-url} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {access_token}" \
  -H 'X-PD-External-User-ID: {your_external_user_id}' \
  -H 'X-PD-Environment: development' \  # 'development' or 'production'
  -d '{
    "message": "Hello, world"
  }'

Configuring workflow steps

When configuring a workflow that’s using your end user’s auth instead of your own, you’ll need to define most configuration fields manually in each step.

For example, normally when you connect your own Google Sheets account directly in the builder, you can dynamically list all of the available sheets from a dropdown.

However, when running workflows on behalf of your end users, that UI configuration doesn’t work, since the Google Sheets account to use is determined at the time of workflow execution. So instead, you’ll need to configure these fields manually.

Either make sure to pass all required configuration data when invoking the workflow, or add a step to your workflow that retrieve it from your database, etc. For example:

curl -X POST https://{your-endpoint-url} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {access_token}" \
  -H 'X-PD-External-User-ID: {your_external_user_id}' \
  -H 'X-PD-Environment: development' \  # 'development' or 'production'
  -d '{ 
    "slackChannel": "#general",
    "messageText": "Hello, world!",
    "gitRepo": "AcmeOrg/acme-repo",
    "issueTitle": "Test Issue"
  }' \

Then in the Slack and GitHub steps, you’d reference those fields directly:

We plan to improve this interface in the future, and potentially allow developers to store end user metadata and configuration data alongside the connected account for your end users, so you won’t need to pass the data at runtime.

Testing workflow steps

To test a step using the connected account of one of your end users in the builder, you’ll need a few things to be configured so that your workflow knows which account to use.

Make sure you have an external user with the relevant connected account(s) saved to your project:

Go to the Users tab in the Connect section of your project to confirm
If not, either connect one from your application or directly in the UI

Pass the environment and external user ID:

Once you’ve added an HTTP trigger to the workflow, click Generate test event

Click on the Headers tab
Make sure x-pd-environment is set (you’ll likely want to development)
Make sure to also pass x-pd-external-user-id with the external user ID of the user you’d like to test with

Troubleshooting

For help debugging issues with your workflow, you can return verbose error messages to the caller by configuring the HTTP trigger to Return a custom response from your workflow.

With that setting enabled on the trigger, below is an example of this error:

curl -X POST https://{your-endpoint-url} \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer {access_token}' \
  -H "x-pd-environment: development" \
  -H "x-pd-external-user-id: abc-123" \
  -d '{
    "slackChannel": "#general",
    "messageText": "Hello, world! (sent via curl)",
    "hubSpotList": "prospects",
    "contactEmail": "foo@example.com"
  }' \
Pipedream Connect Error: Required account for hubspot not found for external user ID abc-123 in development

Common errors

No external user ID passed, but one or more steps require it

One or more steps in the workflow are configured to Use end user’s auth via Connect, but no external user ID was passed when invoking the workflow.
Refer to the docs to make sure you’re passing external user ID correctly when invoking the workflow.

No matching external user ID

There was an external user ID passed, but it didn’t match any users in the project.
Double-check that the external user ID that you passed when invoking the workflow matches one either in the UI or via the API.

Required account not found for external user ID

The external user ID was passed when invoking the workflow, but the user doesn’t have a connected account for one or more of the apps that are configured to use it in this workflow execution.
You can check which connected accounts are available for that user in the UI or via the API.

Running workflows for your users in production requires a higher tier plan

Anyone is able to run workflows for your end users in development. The Business plan is required to run on behalf of production users.
Schedule a call with our sales team and learn more about pricing here.

Known limitations

Workflows can only use a single external user’s auth per execution

Right now you cannot invoke a workflow to loop through many external user IDs within a single execution.
You can only run a workflow for a single external user ID at a time (for now).

The external user ID to use during execution must be passed in the triggering event

You can’t run a workflow on a timer for example, and look up the external user ID to use at runtime.
The external user ID must be passed in the triggering event, typically via HTTP trigger.

Managed auth Running components