Running components for your end users

Pipedream Connect provides you with the ability to pre-configure, deploy and invoke components on behalf of your end users.

What are components?

In the context of Pipedream Connect, components are self-contained executable units of code. They are usually provided with some arbitrary input and produce a result that’s exported as output. These components are developed by the Pipedream community and their source code is available in our public Github repository.

To learn more about components in detail, please visit our Components guide.

Running components for your end users via Pipedream Connect is in beta, and we’re looking for feedback. Please let us know how you’re using it, what’s not working, and what else you’d like to see.

How to run components for your end users

Refer to the Connect API docs for the full API reference. Below is a quickstart with a few specific examples.

Find the app you want to use

You can also skip steps 1 and 2 if you already know the component you want to use or if you’d prefer to pass a natural language prompt to Pipedream’s component search API.

In order to find the right trigger or action to configure and run, you first need to find the app. For this example, we’ll search for gitlab.

GET /v1/connect/apps?q=gitlab

Here’s the response:

{
  "page_info": {
    "total_count": 1,
    "count": 1,
    "start_cursor": "Z2l0bGFi",
    "end_cursor": "Z2l0bGFi"
  },
  "data": [
    {
      "id": "app_1Z2hw1",
      "name_slug": "gitlab",
      "name": "GitLab",
      "auth_type": "oauth",
      "description": "GitLab is the most comprehensive AI-powered DevSecOps Platform.",
      "img_src": "https://assets.pipedream.net/s.v0/app_1Z2hw1/logo/orig",
      "custom_fields_json": "[{\"name\":\"base_api_url\",\"label\":\"Base API URL\",\"description\":\"The Base API URL defaults to `gitlab.com`. If you are using self-hosted Gitlab, enter your base url here, e.g. `gitlab.example.com`\",\"default\":\"gitlab.com\",\"optional\":null}]",
      "categories": [
        "Developer Tools"
      ]
    }
  ]
}

List the available components for the app

Once you have the app you want to use, now you can list the triggers and/or actions for that app. We’ll list the actions for Gitlab and we’ll pass the name_slug gitlab as the app.

GET /actions?app=gitlab

Here’s the response:

{
  "page_info": {
    "total_count": 20,
    "count": 10,
    "start_cursor": "c2NfbHlpRThkQQ",
    "end_cursor": "c2NfQjVpTkJBTA"
  },
  "data": [
    {
      "name": "List Commits",
      "version": "0.0.2",
      "key": "gitlab_developer_app-list-commits"
    },
    {
      "name": "Update Issue",
      "version": "0.0.1",
      "key": "gitlab_developer_app-update-issue"
    },
    {
      "name": "Update Epic",
      "version": "0.0.1",
      "key": "gitlab_developer_app-update-epic"
    },
    {
      "name": "Search Issues",
      "version": "0.0.1",
      "key": "gitlab_developer_app-search-issues"
    },
    {
      "name": "List Repo Branches",
      "version": "0.0.1",
      "key": "gitlab_developer_app-list-repo-branches"
    },
    {
      "name": "Get Repo Branch",
      "version": "0.0.1",
      "key": "gitlab_developer_app-get-repo-branch"
    },
    {
      "name": "Get Issue",
      "version": "0.0.1",
      "key": "gitlab_developer_app-get-issue"
    },
    {
      "name": "Create issue",
      "version": "0.0.1",
      "key": "gitlab_developer_app-create-issue"
    },
    {
      "name": "Create Epic",
      "version": "0.0.1",
      "key": "gitlab_developer_app-create-epic"
    },
    {
      "name": "Create Branch",
      "version": "0.0.1",
      "key": "gitlab_developer_app-create-branch"
    }
  ]
}

Retrieve the component’s definition

In order to run a component for your end users, you need to understand the component’s definition. Now that you have the component’s key from the previous step, you can retrieve its structure from the Pipedream API. See the component structure section in our docs for more details.

As an example, the following API call will return the structure of the List Commits action for Gitlab:

GET /v1/connect/components/gitlab-list-commits

The response will contain the component’s structure, including its (human-understandable) name, version, and most importantly, the configuration options that the component accepts (also known as props or “properties”). Here’s an example of the response for the component in the example above:

{
  "data": {
    "name": "List Commits",
    "version": "0.0.3",
    "key": "gitlab-list-commits",
    "configurable_props": [
      {
        "name": "gitlab",
        "type": "app",
        "app": "gitlab"
      },
      {
        "name": "projectId",
        "type": "integer",
        "label": "Project ID",
        "description": "The project ID, as displayed in the main project page",
        "remoteOptions": true
      },
      {
        "name": "refName",
        "type": "string",
        "label": "Branch Name",
        "description": "The name of the branch",
        "remoteOptions": true,
        "optional": true
      },
      {
        "name": "max",
        "type": "integer",
        "label": "Max Results",
        "description": "Max number of results to return. Default value is `100`",
        "optional": true,
        "default": 100
      }
    ]
  }
}

Using this information, you can now drive the configuration of the component for your end users, as described in the next section.

Configure the component

Component execution on behalf of your end users requires a few preliminary steps, focused on getting the right input parameters (aka props) to the component.

Configuring each prop for a component usually involves an API call to our backend to retrieve the possible values, unless the values that a prop can take are static/free-form. The endpoint is accessible at:

POST /v1/connect/components/configure

Typically, the options for a prop are linked to a specific user’s account. Each of these props implements an options method that retrieves the necessary options from the third-party API, formats them, and sends them back in the response for the end user to select.

The payload for the configuration API call must contain the following fields:

external_user_id: the ID of your user on your end
id: the component’s unique ID (aka key)
prop_name: the name of the prop you want to configure
configured_props: an object containing the props that have already been configured. The initial configuration call must contain the ID of the account (aka authProvisionId) that your user has connected to the target app (see this section for more details on how to create these accounts)

We’ll use the List Commits component for Gitlab as an example, to illustrate a call that retrieves the options for the projectId prop of that component. The payload sent to the configuration API would look like this:

{
  "external_user_id": "demo-34c13d13-a31e-4a3d-8b63-0ac954671095",
  "id": "gitlab-list-commits",
  "prop_name": "projectId",
  "configured_props": {
    "gitlab": {
      "authProvisionId": "apn_oOhaBlD"
    }
  }
}

The response will contain the possible values (and their human-readable labels when applicable) for the prop, as well as any possible errors that might have occurred. The response for the request above would look like this:

{
  "observations": [],
  "context": null,
  "options": [
    {
      "label": "jverce/foo-massive-10231-1",
      "value": 45672541
    },
    {
      "label": "jverce/foo-massive-10231",
      "value": 45672514
    },
    {
      "label": "jverce/foo-massive-14999-2",
      "value": 45672407
    },
    {
      "label": "jverce/foo-massive-14999-1",
      "value": 45672382
    },
    {
      "label": "jverce/foo-massive-14999",
      "value": 45672215
    },
    {
      "label": "jverce/gitlab-development-kit",
      "value": 21220953
    },
    {
      "label": "jverce/gitlab",
      "value": 21208123
    }
  ],
  "errors": [],
  "timings": {
    "api_to_sidekiq": 1734043172355.1042,
    "sidekiq_received": 1734043172357.867,
    "sidekiq_to_lambda": 1734043172363.6812,
    "sidekiq_done": 1734043173461.6406,
    "lambda_configure_prop_called": 1734043172462,
    "lambda_done": 1734043173455
  },
  "stringOptions": null
}

Fields inside configured_props are written in camel case since they refer to the names of props as they appear in the component’s code, they are not attributes that the API itself consumes.

You configure props one-by-one, making a call to the component configuration API for each new prop. Subsequent prop configuration calls will be identical to the one above:

Add the prop you currently want to configure as the prop_name
Include the names and values of all previously-configured props in the configured_props object. Keep this object in your app’s local state, add a prop once you or the end user selects a value, and pass it to the configured_props API param.

For example, to retrieve the configuration options for the refName prop:

{
  "async_handle": "IyvFeE5oNpYd",
  "external_user_id": "demo-34c13d13-a31e-4a3d-8b63-0ac954671095",
  "id": "gitlab-list-commits",
  "prop_name": "refName",
  "configured_props": {
    "gitlab": {
      "authProvisionId": "apn_oOhaBlD"
    },
    "projectId": 21208123
  }
}

Configure dynamic props (optional)

The set of props that a component can accept might not be static, and can change depending on the values of prior props. Props that behave this way are called dynamic props, and they need to be configured in a different way. Props that are dynamic will have a reloadProps attribute set to true in the component’s code.

After configuring a dynamic prop, the set of subsequent props must be recomputed (or reloaded), which is possible using the following API call:

POST /v1/connect/components/props

The payload is similar to the one used for the configuration API, but it excludes the prop_name field since the goal of this call is to reload and retrieve the new set of props, not to configure a specific one.

Using the Add Single Row component for Google Sheets as an example, the request payload would look like this:

{
  "async_handle": "PL41Yf3PuX61",
  "external_user_id": "demo-25092fa8-86c0-4d46-86c9-9dc9bde3b964",
  "id": "google_sheets-add-single-row",
  "configured_props": {
    "googleSheets": {
      "authProvisionId": "apn_V1hMoE7"
    },
    "sheetId": "1BfWjFF2dTW_YDiLISj5N9nKCUErShgugPS434liyytg"
  }
}

In this case, the sheetId prop is dynamic, and so after configuring it, the set of props must be reloaded. The response will contain the new set of props and their definition, similar to when the component information was first retrieved. The response will also contain an ID that can be used to reference the new set of props in subsequent configuration calls. If this is ID is not provided, the set of props will be based on the definition of the component that was retrieved initially.

To illustrate, the response for the request above would look like this:

{
  "observations": [],
  "errors": [],
  "dynamicProps": {
    "id": "dyp_6xUyVgQ",
    "configurableProps": [
      {
        "name": "googleSheets",
        "type": "app",
        "app": "google_sheets"
      },
      {
        "name": "drive",
        "type": "string",
        "label": "Drive",
        "description": "Defaults to `My Drive`. To select a [Shared Drive](https://support.google.com/a/users/answer/9310351) instead, select it from this list.",
        "optional": true,
        "default": "My Drive",
        "remoteOptions": true
      },
      {
        "name": "sheetId",
        "type": "string",
        "label": "Spreadsheet",
        "description": "The Spreadsheet ID",
        "useQuery": true,
        "remoteOptions": true,
        "reloadProps": true
      },
      {
        "name": "worksheetId",
        "type": "string[]",
        "label": "Worksheet(s)",
        "description": "Select a worksheet or enter a custom expression. When referencing a spreadsheet dynamically, you must provide a custom expression for the worksheet.",
        "remoteOptions": true,
        "reloadProps": true
      },
      {
        "name": "hasHeaders",
        "type": "boolean",
        "label": "Does the first row of the sheet have headers?",
        "description": "If the first row of your document has headers, we'll retrieve them to make it easy to enter the value for each column. Note: When using a dynamic reference for the worksheet ID (e.g. `{{steps.foo.$return_value}}`), this setting is ignored.",
        "reloadProps": true
      },
      {
        "name": "myColumnData",
        "type": "string[]",
        "label": "Values",
        "description": "Provide a value for each cell of the row. Google Sheets accepts strings, numbers and boolean values for each cell. To set a cell to an empty value, pass an empty string."
      }
    ]
  }
}

Execution

Once all the props have been configured, the component can be invoked. Pipedream supports two types of components: actions and sources (aka triggers).

Actions are components that perform a task by taking an input either during configuration and/or during invocation (usually both), and produce a result. Sources are very similar, but the difference is that they are not invoked directly by Pipedream users, but rather by events that happen on a third-party service. For example, the “New File” source for Google Drive will be triggered every time a new file is created in a specific folder in Google Drive.

All this means is that actions can be invoked manually on demand, while sources are instead deployed and run automatically when the event they are listening for occurs.

Invoking an action

At the end of the configuration process for an action, you’ll end up with a payload that you can use to invoke the action. The payload is similar to the one used for configuring a prop, with the exception of the prop_name attribute (because we’re not configuring any props at this point):

{
  "async_handle": "xFfBakdTGTkI",
  "external_user_id": "demo-20a1b80e-23a5-4d4d-a8ef-c91cdbd0dad9",
  "id": "gitlab-list-commits",
  "configured_props": {
    "gitlab": {
      "authProvisionId": "apn_JjhWOl0"
    },
    "projectId": 45672541,
    "refName": "main"
  }
}

To run the action with this configuration, simply send it as the request payload to the following endpoint:

POST /v1/connect/actions/run

The output of executing the action will be a JSON object containing the following fields:

exports: all the named exports produced by the action, like when calling $.export in a Node.js component.
os: a list of observations produced by the action (e.g. logs, errors, etc).
ret: the return value of the action, if any.

The following (abbreviated) example shows the output of running the action above:

{
  "exports": {
    "$summary": "Retrieved 1 commit"
  },
  "os": [],
  "ret": [
    {
      "id": "387262aea5d4a6920ac76c1e202bc9fd0841fea5",
      "short_id": "387262ae",
      "created_at": "2023-05-03T03:03:25.000+00:00",
      "parent_ids": [],
      "title": "Initial commit",
      "message": "Initial commit",
      "author_name": "Jay Vercellone",
      "author_email": "nope@pipedream.com",
      "authored_date": "2023-05-03T03:03:25.000+00:00",
      "committer_name": "Jay Vercellone",
      "committer_email": "nope@pipedream.com",
      "committed_date": "2023-05-03T03:03:25.000+00:00",
      "trailers": {},
      "extended_trailers": {},
      "web_url": "https://gitlab.com/jverce/foo-massive-10231-1/-/commit/387262aea5d4a6920ac76c1e202bc9fd0841fea5"
    }
  ]
}

Deploying a source

Because sources are exercised by events that happen on a third-party service, their semantics are different from actions. Once a source is configured, it must be deployed in order to start listening for events. You might be asking where do those events go, and you’d be right to ask. That’s the reason why when deploying a source you must provide a URL where the source will send these events, similar to how a webhook works.

Deploying a source is done by sending a payload similar to the one used for running an action, with the addition of the webhook URL mentioned above. Using the New Issue (Instant) source for Gitlab as an example, the payload would look something like this:

{
  "external_user_id": "jverce",
  "id": "gitlab-new-issue",
  "prop_name": "http",
  "configured_props": {
    "gitlab": {
      "authProvisionId": "apn_kVh9AoD"
    },
    "projectId": 45672541
  },
  "webhook_url": "https://events.example.com/gitlab-new-issue"
}

The endpoint to deploy a source is:

POST /v1/connect/triggers/deploy

If the source deployment succeeds, the response will contain the information regarding the state of the source, including all the component’s props metadata, as well as their values. It will also contain its name, creation date, owner, and most importantly its unique ID, which can be used to manage the source in the future (e.g. delete it). The response for the request above would look like this:

{
  "data": {
    "id": "dc_dAuGmW7",
    "owner_id": "exu_oedidz",
    "component_id": "sc_3vijzQr",
    "configurable_props": [
      {
        "name": "gitlab",
        "type": "app",
        "app": "gitlab"
      },
      {
        "name": "db",
        "type": "$.service.db"
      },
      {
        "name": "http",
        "type": "$.interface.http",
        "customResponse": true
      },
      {
        "name": "projectId",
        "type": "integer",
        "label": "Project ID",
        "description": "The project ID, as displayed in the main project page",
        "remoteOptions": true
      }
    ],
    "configured_props": {
      "gitlab": {
        "authProvisionId": "apn_kVh9AoD"
      },
      "db": {
        "type": "$.service.db"
      },
      "http": {
        "endpoint_url": "https://39643dc2c8ff98a019f3fdc9078d6db9.m.pipedream.net"
      },
      "projectId": 45672541
    },
    "active": true,
    "created_at": 1734028283,
    "updated_at": 1734028283,
    "name": "My first project - exu_oedidz",
    "name_slug": "my-first-project---exu-oedidz-2"
  }
}

In the example above, the source ID is dc_dAuGmW7, which can be used to delete the source in the future:

DELETE /v1/connect/deployed-triggers/dc_dAuGmW7?external_user_id=jverce

Running workflows API & SDK reference