Running workflows for your end users

Just like you can build and run internal workflows for your team, you can run workflows for your end users, too.

Whether you’re building well-defined integrations or autonomous AI agents, workflows provide a powerful set of tools for running code or pre-defined actions on behalf of your users. Pipedream’s UI makes it easy to build, test, and debug workflows.

What are workflows?

Workflows are sequences of steps triggered by an event, like an HTTP request, or new rows in a Google sheet.

You can use pre-built actions or custom Node.js, Python, Golang, or Bash code in workflows and connect to any of our 2,700 integrated apps.

Workflows also have built-in:

Read the quickstart to learn more.

Getting started

Create a workflow

Create a new workflow or open an existing one.

Add an HTTP trigger

To get started building workflows for your end users:

Add an HTTP trigger to your workflow
Generate a test event with the required headers:
- x-pd-environment: development
- x-pd-external-user-id: {your_external_user_id}

See the Triggering your workflow section below for details on securing your workflow with OAuth and deploying triggers on behalf of your end users.

Configure accounts to use your end users’ auth

When you configure pre-built actions or custom code that connects to third-party APIs, you can link accounts in one of two ways:

Use your own account: If you’re connecting to an API that uses your own API key or developer account — for example, a workflow that connects to the OpenAI API or a PostgreSQL database — click the Connect account button to link your own, static account.

Use your end users’ auth: If you’re building a workflow that connects to your end users’ accounts — for example, a workflow that sends a message with your user’s Slack account — you can select the option to Use end user’s auth via Connect:

When you trigger the workflow, Pipedream will look up the corresponding account for the end user whose user ID you provide when invoking the workflow.

Connect a test account

To run an end-to-end test as an end user, you need to have users and connected accounts in your project. If you already have a development account linked, you can skip this step.

If you don’t, the fastest way to do this is on the Users tab in your Pipedream project:

You’ll see there’s a button to Connect account
Go through the flow and make sure to create the account in development mode
Note the external user ID of the account you just connected, you’ll need it in the next step

Generate a test request

Test events are critical for developing workflows effectively. Without a test event, you won’t be able to test your workflow end to end in the builder, see the shape of the event data that triggers the workflow, and the lookup to use your end user’s auth won’t work.

To generate a test event, click Send Test Event in the trigger, and fill in the event data. This will trigger the workflow and allow you to test the workflow end to end in the builder.

Make sure to include these headers in your test request:

x-pd-environment: development
x-pd-external-user-id: {your_external_user_id}

Deploy your workflow

When you’re done with the workflow, click Deploy at the top right.

Invoke the workflow

If you’re using TypeScript or a JavaScript runtime, install the Pipedream SDK. Pipedream also provides an HTTP API for invoking workflows (see example below).

npm i @pipedream/sdk

To invoke workflows, you’ll need:

The OAuth client ID and secret from your OAuth client in step 2 above (if configured)
Your Project ID
Your workflow’s HTTP endpoint URL
The external user ID of the user you’d like to run the workflow for
The Connect environment tied to the user’s account

Then invoke the workflow like so:

import { createBackendClient, HTTPAuthType } from "@pipedream/sdk/server";
 
// These secrets should be saved securely and passed to your environment
const pd = createBackendClient({
  environment: "development", // change to production if running for a test production account, or in production
  credentials: {
    clientId: "{oauth_client_id}",
    clientSecret: "{oauth_client_secret}",
  },
  projectId: "{your_project_id}"
});
 
await pd.invokeWorkflowForExternalUser(
  "{your_endpoint_url}", // pass the endpoint ID or full URL here
  "{your_external_user_id}" // The end user's ID in your system
  {
    method: "POST",
    body: {
      message: "Hello World"
    }
  },
  HTTPAuthType.OAuth // Will automatically send the Authorization header with a fresh token
)

import { createBackendClient } from "@pipedream/sdk/server";
 
// These secrets should be saved securely and passed to your environment
const client = createBackendClient({
  environment: "development", // change to production if running for a test production account, or in production
  credentials: {
    clientId: "{oauth_client_id}",
    clientSecret: "{oauth_client_secret}"
  },
  projectId: "{your_project_id}"
});
 
const response = await client.invokeWorkflowForExternalUser(
  "{your_endpoint_url}", // pass the endpoint ID or full URL here
  "{your_external_user_id}" // The end user's ID in your system
  {
    method: "POST",
    body: {
      message: "Hello World"
    }
  }
)

# First, obtain an OAuth access token
curl -X POST https://api.pipedream.com/v1/oauth/token \
  -H "Content-Type: application/json" \
  -d '{
    "grant_type": "client_credentials",
    "client_id": "{oauth_client_id}",
    "client_secret": "{oauth_client_secret}"
  }'
 
# The response will include an access_token. Use it in the Authorization header below.
 
curl -X POST https://{your-endpoint-url} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {access_token}" \
  -H 'X-PD-External-User-ID: {your_external_user_id}' \
  -H 'X-PD-Environment: development' \  # 'development' or 'production'
  -d '{
    "message": "Hello, world"
  }'

Configuring workflow steps

When configuring a workflow that’s using your end user’s auth instead of your own, you’ll need to define most configuration fields manually in each step.

For example, normally when you connect your own Google Sheets account directly in the builder, you can dynamically list all of the available sheets from a dropdown.

However, when running workflows on behalf of your end users, that UI configuration doesn’t work, since the Google Sheets account to use is determined at the time of workflow execution. So instead, you’ll need to configure these fields manually.

Either make sure to pass all required configuration data when invoking the workflow, or add a step to your workflow that retrieve it from your database, etc. For example:

curl -X POST https://{your-endpoint-url} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {access_token}" \
  -H 'X-PD-External-User-ID: {your_external_user_id}' \
  -H 'X-PD-Environment: development' \  # 'development' or 'production'
  -d '{
    "slackChannel": "#general",
    "messageText": "Hello, world!",
    "gitRepo": "AcmeOrg/acme-repo",
    "issueTitle": "Test Issue"
  }' \

Then in the Slack and GitHub steps, you’d reference those fields directly:

We plan to improve this interface in the future, and potentially allow developers to store end user metadata and configuration data alongside the connected account for your end users, so you won’t need to pass the data at runtime. Let us know if that’s a feature you’d like to see.

Testing

To test a step using the connected account of one of your end users in the builder, you’ll need a few things to be configured so that your workflow knows which account to use.

Make sure you have an external user with the relevant connected account(s) saved to your project:

Go to the Users tab in the Connect section of your project to confirm
If not, either connect one from your application or directly in the UI

Pass the environment and external user ID:

Once you’ve added an HTTP trigger to the workflow, click Generate test event

Click on the Headers tab
Make sure x-pd-environment is set (you’ll likely want to development)
Make sure to also pass x-pd-external-user-id with the external user ID of the user you’d like to test with

Triggering your workflow

You have two options for triggering workflows that run on behalf of your end users:

Invoke via HTTP webhook
Deploy an event source (Slack, Gmail, etc.)

HTTP Webhook

The most common way to trigger workflows is via HTTP webhook. We strongly recommend creating a Pipedream OAuth client and authenticating inbound requests to your workflows.

This section refers to authenticating requests to the Pipedream API. For info on how managed auth works for your end users, refer to the managed auth quickstart.

To get started, you’ll need:

OAuth client ID and secret for authenticating with the Pipedream API
Your project ID
Your workflow’s HTTP endpoint URL
The external user ID of your end user
The Connect environment

import { createBackendClient, HTTPAuthType } from "@pipedream/sdk/server";
 
// These secrets should be saved securely and passed to your environment
const pd = createBackendClient({
  environment: "development", // change to production if running for a test production account, or in production
  credentials: {
    clientId: "{oauth_client_id}",
    clientSecret: "{oauth_client_secret}",
  },
  projectId: "{your_project_id}"
});
 
await pd.invokeWorkflowForExternalUser(
  "{your_endpoint_url}", // pass the endpoint ID or full URL here
  "{your_external_user_id}" // The end user's ID in your system
  {
    method: "POST",
    body: {
      message: "Hello World"
    }
  },
  HTTPAuthType.OAuth // Will automatically send the Authorization header with a fresh token
)

import { createBackendClient } from "@pipedream/sdk/server";
 
// These secrets should be saved securely and passed to your environment
const client = createBackendClient({
  environment: "development", // change to production if running for a test production account, or in production
  credentials: {
    clientId: "{oauth_client_id}",
    clientSecret: "{oauth_client_secret}"
  },
  projectId: "{your_project_id}"
});
 
const response = await client.invokeWorkflowForExternalUser(
  "{your_endpoint_url}", // pass the endpoint ID or full URL here
  "{your_external_user_id}" // The end user's ID in your system
  {
    method: "POST",
    body: {
      message: "Hello World"
    }
  }
)

# First, obtain an OAuth access token
curl -X POST https://api.pipedream.com/v1/oauth/token \
  -H "Content-Type: application/json" \
  -d '{
    "grant_type": "client_credentials",
    "client_id": "{oauth_client_id}",
    "client_secret": "{oauth_client_secret}"
  }'
 
# The response will include an access_token. Use it in the Authorization header below.
 
curl -X POST https://{your-endpoint-url} \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer {access_token}" \
  -H 'X-PD-External-User-ID: {your_external_user_id}' \
  -H 'X-PD-Environment: development' \  # 'development' or 'production'
  -d '{
    "message": "Hello, world"
  }'

Deploy an event source

You can programmatically deploy triggers via the API to have events from integrated apps (like new Slack messages or new emails in Gmail) trigger your workflow. This allows you to:

Deploy triggers for specific users from your application
Configure trigger parameters per-user
Manage deployed triggers via the API

See the API documentation for detailed examples of deploying and managing triggers.

OAuth client requirements

When using OAuth apps (like Google Drive, Slack, Notion, etc.) with your end users, you must use your own custom OAuth clients.

Register your own OAuth application with each third-party service (Google Drive, Slack, etc.)
Add your OAuth client credentials to Pipedream
Make sure to include your oauthAppId when connecting accounts for your end users

For detailed instructions, see the OAuth Clients documentation.

Troubleshooting

For help debugging issues with your workflow, you can return verbose error messages to the caller by configuring the HTTP trigger to Return a custom response from your workflow.

With that setting enabled on the trigger, below is an example of this error:

curl -X POST https://{your-endpoint-url} \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer {access_token}' \
  -H "x-pd-environment: development" \
  -H "x-pd-external-user-id: abc-123" \
  -d '{
    "slackChannel": "#general",
    "messageText": "Hello, world! (sent via curl)",
    "hubSpotList": "prospects",
    "contactEmail": "foo@example.com"
  }' \
Pipedream Connect Error: Required account for hubspot not found for external user ID abc-123 in development

Common errors

No external user ID passed, but one or more steps require it

One or more steps in the workflow are configured to Use end user’s auth via Connect, but no external user ID was passed when invoking the workflow.
Refer to the docs to make sure you’re passing external user ID correctly when invoking the workflow.

No matching external user ID

There was an external user ID passed, but it didn’t match any users in the project.
Double-check that the external user ID that you passed when invoking the workflow matches one either in the UI or via the API.

Required account not found for external user ID

The external user ID was passed when invoking the workflow, but the user doesn’t have a connected account for one or more of the apps that are configured to use it in this workflow execution.
You can check which connected accounts are available for that user in the UI or via the API.

Running workflows for your users in production requires a higher tier plan

Anyone is able to run workflows for your end users in development.
Visit the pricing page for the latest info on using Connect in production.

Known limitations

Workflows can only use a single external user’s auth per execution

Right now you cannot invoke a workflow to loop through many external user IDs within a single execution.
You can only run a workflow for a single external user ID at a time (for now).

The external user ID to use during execution must be passed in the triggering event

You can’t run a workflow on a timer for example, and look up the external user ID to use at runtime.
The external user ID must be passed in the triggering event, typically via HTTP trigger.

Cannot use multiple accounts for the same app during a single execution

If a user has multiple accounts for the same app (tied to a single external user), Pipedream will use the most recently created account.
Learn about managing connected accounts for your end users.

API Proxy API & SDK Reference