' https://myendpoint.m.pipedream.net
```
You can use the Pipedream SDK to automatically refresh access tokens and invoke workflows, or make HTTP requests directly to the workflow’s URL:
```javascript JavaScript theme={null}
import { PipedreamClient } from "@pipedream/sdk";
// These secrets should be saved securely and passed to your environment
const client = new PipedreamClient({
clientId: "{oauth_client_id}",
clientSecret: "{oauth_client_secret}",
projectId: "{project_id}",
projectEnvironment: "development" // or "production"
});
await client.workflows.invokeForExternalUser(
"enabc123", // pass the endpoint ID or full URL here
"{external_user_id}", // The end user's ID in your system
"POST", // HTTP method
{
key: "value",
} // request body
)
```
```bash cURL theme={null}
# First, obtain an OAuth access token
curl -X POST https://api.pipedream.com/v1/oauth/token \
-H "Content-Type: application/json" \
-d '{
"grant_type": "client_credentials",
"client_id": "{oauth_client_id}",
"client_secret": "{oauth_client_secret}"
}'
# The response will include an access_token. Use it in the Authorization header below.
curl -X POST https://{your-endpoint-url} \
-H "Content-Type: application/json" \
-H "Authorization: Bearer {access_token}" \
-d '{
"message": "Hello, world"
}'
```
#### Implement your own authorization logic
Since you have access to the entire request object, and can issue any HTTP response from a workflow, you can implement custom logic to validate requests.
For example, you could require JWT tokens and validate those tokens using the [`jsonwebtoken` package](https://www.npmjs.com/package/jsonwebtoken) at the start of your workflow.
### Custom domains
To configure endpoints on your own domain, e.g. `endpoint.yourdomain.com` instead of the default `*.m.pipedream.net` domain, see the [custom domains](/docs/workflows/domains/) docs.
### How Pipedream handles JSON payloads
When you send JSON in the HTTP payload, or when JSON data is sent in the payload from a webhook provider, **Pipedream converts that JSON to its equivalent JavaScript object**. The trigger data can be referenced using [the `steps` object](/docs/workflows/building-workflows/triggers/#shape-of-the-stepstriggerevent-object).
In the [Inspector](/docs/workflows/building-workflows/inspect/), we present `steps.trigger.event` cleanly, indenting nested properties, to make the payload easy to read. Since `steps.trigger.event` is a JavaScript object, it’s easy to reference and manipulate properties of the payload using dot-notation.
### How Pipedream handles `multipart/form-data`
When you send [form data](https://ec.haxx.se/http/http-multipart) to Pipedream using a `Content-Type` of `multipart/form-data`, Pipedream parses the payload and converts it to a JavaScript object with a property per form field. For example, if you send a request with two fields:
```bash theme={null}
curl -F 'name=Leia' -F 'title=General' https://myendpoint.m.pipedream.net
```
Pipedream will convert that to a JavaScript object, `event.body`, with the following shape:
```javascript theme={null}
{
name: "Leia",
title: "General",
}
```
### How Pipedream handles HTTP headers
HTTP request headers will be available in the `steps.trigger.event.headers` steps export in your downstream steps.
Pipedream will automatically lowercase header keys for consistency.
### Pipedream-specific request parameters
These params can be set as headers or query string parameters on any request to a Pipedream HTTP endpoint.
#### `x-pd-nostore`
Set to `1` to prevent logging any data for this execution. Pipedream will execute all steps of the workflow, but no data will be logged to Pipedream. No event will show up in the inspector or the Event History UI.
If you need to disable logging for *all* requests, use the workflow’s [Data Retention controls](/docs/workflows/building-workflows/settings/#data-retention-controls), instead.
#### `x-pd-notrigger`
Set to `1` to send an event to the workflow for testing. Pipedream will **not** trigger the production version of the workflow, but will display the event in the [list of test events](/docs/workflows/building-workflows/triggers/#selecting-a-test-event) on the HTTP trigger.
#### Limits
You can send any content, up to the [HTTP payload size limit](/docs/workflows/limits/#http-request-body-size), as a part of the form request. The content of uploaded images or other binary files does not contribute to this limit — the contents of the file will be uploaded at a Pipedream URL you have access to within your source or workflow. See the section on [Large File Support](/docs/workflows/building-workflows/triggers/#large-file-support) for more detail.
### Sending large payloads
*If you’re uploading files, like images or videos, you should use the [large file upload interface](/docs/workflows/building-workflows/triggers/#large-file-support), instead*.
By default, the body of HTTP requests sent to a source or workflow is limited to . **But you can send an HTTP payload of any size to a [workflow](/docs/workflows/building-workflows/) or an [event source](/docs/workflows/building-workflows/triggers/) by including the `pipedream_upload_body=1` query string or an `x-pd-upload-body: 1` HTTP header in your request**.
```bash theme={null}
curl -d '{ "name": "Yoda" }' \
https://endpoint.m.pipedream.net\?pipedream_upload_body\=1
curl -d '{ "name": "Yoda" }' \
-H "x-pd-upload-body: 1" \
https://endpoint.m.pipedream.net
```
In workflows, Pipedream saves the raw payload data in a file whose URL you can reference in the variable `steps.trigger.event.body.raw_body_url`.
![]()
Within your workflow, you can download the contents of this data using the **Send HTTP Request** action, or [by saving the data as a file to the `/tmp` directory](/docs/workflows/building-workflows/code/nodejs/working-with-files/).
#### Example: Download the HTTP payload using the Send HTTP Request action
*Note: you can only download payloads at most* *in size using this method. Otherwise, you may encounter a [Function Payload Limit Exceeded](/docs/troubleshooting/#function-payload-limit-exceeded) error.*
You can download the large HTTP payload using the **Send HTTP Request** action. [Copy this workflow to see how this works](https://pipedream.com/new?h=tch_egfAby).
The payload from the trigger of the workflow is exported to the variable `steps.retrieve_large_payload.$return_value`:
![]()
#### Example: Download the HTTP payload to the `/tmp` directory
[This workflow](https://pipedream.com/new?h=tch_5ofXkX) downloads the HTTP payload, saving it as a file to the [`/tmp` directory](/docs/workflows/building-workflows/code/nodejs/working-with-files/#the-tmp-directory).
```javascript theme={null}
import stream from "stream";
import { promisify } from "util";
import fs from "fs";
import got from "got";
export default defineComponent({
async run({ steps, $ }) {
const pipeline = promisify(stream.pipeline);
await pipeline(
got.stream(steps.trigger.event.body.raw_body_url),
fs.createWriteStream(`/tmp/raw_body`)
);
},
})
```
You can [read this file](/docs/workflows/building-workflows/code/nodejs/working-with-files/#reading-a-file-from-tmp) in subsequent steps of your workflow.
#### How the payload data is saved
Your raw payload is saved to a Pipedream-owned [Amazon S3 bucket](https://aws.amazon.com/s3/). Pipedream generates a [signed URL](https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) that allows you to access to that file for up to 30 minutes. After 30 minutes, the signed URL will be invalidated, and the file will be deleted.
#### Limits
**You can upload payloads up to 5TB in size**. However, payloads that large may trigger [other Pipedream limits](/docs/workflows/limits/). Please [reach out](https://pipedream.com/support/) with any specific questions or issues.
### Large File Support
*This interface is best used for uploading large files, like images or videos. If you’re sending JSON or other data directly in the HTTP payload, and encountering a **Request Entity Too Large** error, review the section above for [sending large payloads](/docs/workflows/building-workflows/triggers/#sending-large-payloads)*.
You can upload any file to a [workflow](/docs/workflows/building-workflows/) or an [event source](/docs/workflows/building-workflows/triggers/) by making a `multipart/form-data` HTTP request with the file as one of the form parts. **Pipedream saves that file to a Pipedream-owned [Amazon S3 bucket](https://aws.amazon.com/s3/), generating a [signed URL](https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) that allows you to access to that file for up to 30 minutes**. After 30 minutes, the signed URL will be invalidated, and the file will be deleted.
In workflows, these file URLs are provided in the `steps.trigger.event.body` variable, so you can download the file using the URL within your workflow, or pass the URL on to another third-party system for it to process.
![]()
Within your workflow, you can download the contents of this data using the **Send HTTP Request** action, or [by saving the data as a file to the `/tmp` directory](/docs/workflows/building-workflows/code/nodejs/working-with-files/).
#### Example: upload a file using `cURL`
For example, you can upload an image to a workflow using `cURL`:
```bash theme={null}
curl -F 'image=@my_image.png' https://myendpoint.m.pipedream.net
```
The `-F` tells `cURL` we’re sending form data, with a single “part”: a field named `image`, with the content of the image as the value (the `@` allows `cURL` to reference a local file).
When you send this image to a workflow, Pipedream [parses the form data](/docs/workflows/building-workflows/triggers/#how-pipedream-handles-multipartform-data) and converts it to a JavaScript object, `event.body`. Select the event from the [inspector](/docs/workflows/building-workflows/inspect/#the-inspector), and you’ll see the `image` property under `event.body`:
![]()
When you upload a file as a part of the form request, Pipedream saves it to a Pipedream-owned [Amazon S3 bucket](https://aws.amazon.com/s3/), generating a [signed URL](https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) that allows you to access to that file for up to 30 minutes. After 30 minutes, the signed URL will be invalidated, and the file will be deleted.
Within the `image` property of `event.body`, you’ll see the value of this URL in the `url` property, along with the `filename` and `mimetype` of the file. Within your workflow, you can download the file, or pass the URL to a third party system to handle, and more.
#### Example: Download this file to the `/tmp` directory
[This workflow](https://pipedream.com/@dylburger/example-download-an-image-to-tmp-p_KwC2Ad/edit) downloads an image passed in the `image` field in the form request, saving it to the [`/tmp` directory](/docs/workflows/building-workflows/code/nodejs/working-with-files/#the-tmp-directory).
```javascript theme={null}
import stream from "stream";
import { promisify } from "util";
import fs from "fs";
import got from "got";
const pipeline = promisify(stream.pipeline);
await pipeline(
got.stream(steps.trigger.event.body.image.url),
fs.createWriteStream(`/tmp/${steps.trigger.event.body.image.filename}`)
);
```
#### Example: Upload image to your own Amazon S3 bucket
[This workflow](https://pipedream.com/@dylburger/example-save-uploaded-file-to-amazon-s3-p_o7Cm9z/edit) streams the uploaded file to an Amazon S3 bucket you specify, allowing you to save the file to long-term storage.
#### Limits
Since large files are uploaded using a `Content-Type` of `multipart/form-data`, the limits that apply to [form data](/docs/workflows/building-workflows/triggers/#how-pipedream-handles-multipartform-data) also apply here.
The content of the file itself does not contribute to the HTTP payload limit imposed for forms. **You can upload files up to 5TB in size**. However, files that large may trigger [other Pipedream limits](/docs/workflows/limits/). Please [reach out](https://pipedream.com/support/) with any specific questions or issues.
### Cross-Origin HTTP Requests
We return the following headers on HTTP `OPTIONS` requests:
```http theme={null}
Access-Control-Allow-Origin: *
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
```
Thus, your endpoint will accept [cross-origin HTTP requests](https://developer.mozilla.org/en-US/Web/HTTP/CORS) from any domain, using any standard HTTP method.
### HTTP Responses
#### Default HTTP response
By default, when you send a [valid HTTP request](/docs/workflows/building-workflows/triggers/#valid-requests) to your endpoint URL, you should expect to receive a `200 OK` status code with the following payload:
```html theme={null}
Success!
To customize this response, check out our docs here
```
When you’re processing HTTP requests, you often don’t need to issue any special response to the client. We issue this default response so you don’t have to write any code to do it yourself.
**How can my workflow run faster?**
See [our guide on running workflows faster](/docs/troubleshooting/#how-can-my-workflow-run-faster).
#### Customizing the HTTP response
If you need to issue a custom HTTP response from a workflow, you can either:
* Use the **Return HTTP response** action, available on the **HTTP / Webhook** app, or
* **Use the `$.respond()` function in a Code or Action step**.
#### Using the HTTP Response Action
The HTTP Response action lets you return HTTP responses without the need to write code. You can customize the response status code, and optionally specify response headers and body.
This action uses `$.respond()` and will always [respond immediately](/docs/workflows/building-workflows/triggers/#returning-a-response-immediately) when called in your workflow. A [response error](/docs/workflows/building-workflows/triggers/#errors-with-http-responses) will still occur if your workflow throws an Error before this action runs.
#### Using custom code with `$.respond()`
You can return HTTP responses in Node.js code with the `$.respond()` function.
`$.respond()` takes a single argument: an object with properties that specify the body, headers, and HTTP status code you’d like to respond with:
```javascript theme={null}
defineComponent({
async run({ steps, $ }) {
await $.respond({
status: 200,
headers: { "my-custom-header": "value" },
body: { message: "My custom response" }, // This can be any string, object, Buffer, or Readable stream
});
},
});
```
The value of the `body` property can be either a string, object, a [Buffer](https://nodejs.org/api/buffer.html#buffer_buffer) (binary data), or a [Readable stream](https://nodejs.org/api/stream.html#stream_readable_streams). Attempting to return any other data may yield an error.
In the case where you return a Readable stream:
* You must `await` the `$.respond` function (`await $.respond({ ... }`)
* The stream must close and be finished reading within your [workflow execution timeout](/docs/workflows/limits/#time-per-execution).
* You cannot return a Readable and use the [`immediate: true`](/docs/workflows/building-workflows/triggers/#returning-a-response-immediately) property of `$.respond`.
#### Timing of `$.respond()` execution
You may notice some response latency calling workflows that use `$.respond()` from your HTTP client. By default, `$.respond()` is called at the end of your workflow, after all other code is done executing, so it may take some time to issue the response back.
If you need to issue an HTTP response in the middle of a workflow, see the section on [returning a response immediately](/docs/workflows/building-workflows/triggers/#returning-a-response-immediately).
#### Returning a response immediately
You can issue an HTTP response within a workflow, and continue the rest of the workflow execution, by setting the `immediate` property to `true`:
```javascript theme={null}
defineComponent({
async run({ steps, $ }) {
await $.respond({
immediate: true,
status: 200,
headers: { "my-custom-header": "value" },
body: { message: "My custom response" },
});
},
});
```
Passing `immediate: true` tells `$.respond()` to issue a response back to the client at this point in the workflow. After the HTTP response has been issued, the remaining code in your workflow runs.
This can be helpful, for example, when you’re building a Slack bot. When you send a message to a bot, Slack requires a `200 OK` response be issued immediately, to confirm receipt:
```javascript theme={null}
defineComponent({
async run({ steps, $ }) {
await $.respond({
immediate: true,
status: 200,
body: "",
});
},
});
```
Once you issue the response, you’ll probably want to process the message from the user and respond back with another message or data requested by the user.
[Here’s an example workflow](https://pipedream.com/@dylburger/issue-http-response-immediately-continue-running-workflow-p_pWCWGJ) that shows how to use `immediate: true` and run code after the HTTP response is issued.
#### Errors with HTTP Responses
If you use `$.respond()` in a workflow, **you must always make sure `$.respond()` is called in your code**. If you make an HTTP request to a workflow, and run code where `$.respond()` is *not* called, your endpoint URL will issue a `400 Bad Request` error with the following body:
```txt theme={null}
No $.respond called in workflow
```
This might happen if:
* You call `$.respond()` conditionally, where it does not run under certain conditions.
* Your workflow throws an Error before you run `$.respond()`.
* You return data in the `body` property that isn’t a string, object, or Buffer.
If you can’t handle the `400 Bad Request` error in the application calling your workflow, you can implement `try` / `finally` logic to ensure `$.respond()` always gets called with some default message. For example:
```javascript theme={null}
defineComponent({
async run({ steps, $ }) {
try {
// Your code here that might throw an exception or not run
throw new Error("Whoops, something unexpected happened.");
} finally {
await $.respond({
status: 200,
body: {
msg: "Default response",
},
});
}
},
});
```
### Errors
Occasionally, you may encounter errors when sending requests to your endpoint:
#### Request Entity Too Large
The endpoint will issue a `413 Payload Too Large` status code when the body of your request exceeds .
In this case, the request will still appear in the inspector, with information on the error.
#### API key does not exist
Your API key is the host part of the endpoint, e.g. the `eniqtww30717` in `eniqtww30717.m.pipedream.net`. If you attempt to send a request to an endpoint that does not exist, we’ll return a `404 Not Found` error.
We’ll also issue a 404 response on workflows with an HTTP trigger that have been disabled.
#### Too Many Requests
If you send too many requests to your HTTP source within a small period of time, we may issue a `429 Too Many Requests` response. [Review our limits](/docs/workflows/limits/) to understand the conditions where you might be throttled.
You can also [reach out](https://pipedream.com/support/) to inquire about raising this rate limit.
If you control the application sending requests, you should implement [a backoff strategy](https://medium.com/clover-platform-blog/conquering-api-rate-limiting-dcac5552714d) to temporarily slow the rate of events.
## Schedule
Pipedream allows you to run hosted scheduled jobs — commonly-referred to as a “cron job” — [for free](/docs/pricing/). You can think of workflows like scripts that run on a schedule.
You can write scheduled job to send an HTTP request, send a scheduled email, run any Node.js or Python code, connect to any API, and much more. Pipedream manages the servers where these jobs run, so you don’t have to worry about setting up a server of your own or operating some service just to run code on a schedule. You write the workflow, we take care of the rest.
### Choosing a Schedule trigger
To create a new scheduled job, create a new workflow and search for the **Schedule** trigger:
![]()
By default, your trigger will be turned **Off**. **To enable it, select either of the scheduling options**:
* **Every** : run the job every N days, hours, minutes (e.g. every 1 day, every 3 hours).
* **Cron Expression** : schedule your job using a cron expression. For example, the expression `0 0 * * *` will run the job every day at midnight. Cron expressions can be tied to any timezone.
### Testing a scheduled job
If you’re running a scheduled job once a day, you probably don’t want to wait until the next day’s run to test your new code. You can manually run the workflow associated with a scheduled job at any time by pressing the **Run Now** button.
### Job History
You’ll see the history of job executions under the **Job History** section of the [Inspector](/docs/workflows/building-workflows/inspect/).
Clicking on a specific job shows the execution details for that job — all the logs and observability associated with that run of the workflow.
### Trigger a notification to an external service (email, Slack, etc.)
You can send yourself a notification — for example, an email or a Slack message — at any point in a workflow by using the relevant [Action](/docs/components/contributing/#actions) or [Destination](/docs/workflows/data-management/destinations/).
If you’d like to email yourself when a job finishes successfully, you can use the [Email Destination](/docs/workflows/data-management/destinations/email/). You can send yourself a Slack message using the Slack Action, or trigger an [HTTP request](/docs/workflows/data-management/destinations/http/) to an external service.
You can also [write code](/docs/workflows/building-workflows/code/) to trigger any complex notification logic you’d like.
### Troubleshooting your scheduled jobs
When you run a scheduled job, you may need to troubleshoot errors or other execution issues. Pipedream offers built-in, step-level logs that show you detailed execution information that should aid troubleshooting.
Any time a scheduled job runs, you’ll see a new execution appear in the [Inspector](/docs/workflows/building-workflows/inspect/). This shows you when the job ran, how long it took to run, and any errors that might have occurred. **Click on any of these lines in the Inspector to view the details for a given run**.
Code steps show [logs](/docs/workflows/building-workflows/code/nodejs/#logs) below the step itself. Any time you run `console.log()` or other functions that print output, you should see the logs appear directly below the step where the code ran.
[Actions](/docs/components/contributing/#actions) and [Destinations](/docs/workflows/data-management/destinations/) also show execution details relevant to the specific Action or Destination. For example, when you use the [HTTP Destination](/docs/workflows/data-management/destinations/http/) to make an HTTP request, you’ll see the HTTP request and response details tied to that Destination step:
## Email
When you select the **Email** trigger:
![]()
Pipedream creates an email address specific to your workflow. Any email sent to this address triggers your workflow:
![]()
As soon as you send an email to the workflow-specific address, Pipedream parses its body, headers, and attachments into a JavaScript object it exposes in the `steps.trigger.event` variable that you can access within your workflow. This transformation can take a few seconds to perform. Once done, Pipedream will immediately trigger your workflow with the transformed payload.
[Read more about the shape of the email trigger event](/docs/workflows/building-workflows/triggers/#email).
### Sending large emails
By default, you can send emails up to in total size (content, headers, attachments). Emails over this size will be rejected, and you will not see them appear in your workflow.
**You can send emails up to `{EMAIL_PAYLOAD_SIZE_LIMIT}` in size by sending emails to `[YOUR EMAIL ENDPOINT]@upload.pipedream.net`**. If your workflow-specific email address is `endpoint@pipedream.net`, your “large email address” is `endpoint@upload.pipedream.net`.
Emails delivered to this address are uploaded to a private URL you have access to within your workflow, at the variable `steps.trigger.event.mail.content_url`. You can download and parse the email within your workflow using that URL. This content contains the *raw* email. Unlike the standard email interface, you must parse this email on your own - see the examples below.
#### Example: Download the email using the Send HTTP Request action
*Note: you can only download emails at most* *in size using this method. Otherwise, you may encounter a [Function Payload Limit Exceeded](/docs/troubleshooting/#function-payload-limit-exceeded) error.*
You can download the email using the **Send HTTP Request** action. [Copy this workflow to see how this works](https://pipedream.com/new?h=tch_1AfMyl).
This workflow also parses the contents of the email and exposes it as a JavaScript object using the [`mailparser` library](https://nodemailer.com/extras/mailparser/):
```javascript theme={null}
import { simpleParser } from "mailparser";
export default defineComponent({
async run({ steps, $ }) {
return await simpleParser(steps.get_large_email_content.$return_value);
},
});
```
#### Example: Download the email to the `/tmp` directory, read it and parse it
[This workflow](https://pipedream.com/new?h=tch_jPfaEJ) downloads the email, saving it as a file to the [`/tmp` directory](/docs/workflows/building-workflows/code/nodejs/working-with-files/#the-tmp-directory). Then it reads the same file (as an example), and parses it using the [`mailparser` library](https://nodemailer.com/extras/mailparser/):
```javascript theme={null}
import stream from "stream";
import { promisify } from "util";
import fs from "fs";
import got from "got";
import { simpleParser } from "mailparser";
// To use previous step data, pass the `steps` object to the run() function
export default defineComponent({
async run({ steps, $ }) {
const pipeline = promisify(stream.pipeline);
await pipeline(
got.stream(steps.trigger.event.mail.content_url),
fs.createWriteStream(`/tmp/raw_email`)
);
// Now read the file and parse its contents into the `parsed` variable
// See https://nodemailer.com/extras/mailparser/ for parsing options
const f = fs.readFileSync(`/tmp/raw_email`);
return await simpleParser(f);
},
});
```
#### How the email is saved
Your email is saved to a Pipedream-owned [Amazon S3 bucket](https://aws.amazon.com/s3/). Pipedream generates a [signed URL](https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html) that allows you to access to that file for up to 30 minutes. After 30 minutes, the signed URL will be invalidated, and the file will be deleted.
### Email attachments
You can attach any files to your email, up to [the total email size limit](/docs/workflows/limits/#email-triggers).
Attachments are stored in `steps.trigger.event.attachments`, which provides an array of attachment objects. Each attachment in that array exposes key properties:
* `contentUrl`: a URL that hosts your attachment. You can [download this file to the `/tmp` directory](/docs/workflows/building-workflows/code/nodejs/http-requests/#download-a-file-to-the-tmp-directory) and process it in your workflow.
* `content`: If the attachment contains text-based content, Pipedream renders the attachment in `content`, up to 10,000 bytes.
* `contentTruncated`: `true` if the attachment contained text-based content larger than 10,000 bytes. If `true`, the data in `content` will be truncated, and you should fetch the full attachment from `contentUrl`.
### Appending metadata to the incoming email address with `+data`
Pipedream provides a way to append metadata to incoming emails by adding a `+` sign to the incoming email key, followed by any arbitrary string:
```txt theme={null}
myemailaddr+test@pipedream.net
```
Any emails sent to your workflow-specific email address will resolve to that address, triggering your workflow, no matter the data you add after the `+` sign. Sending an email to both of these addresses triggers the workflow with the address `myemailaddr@pipedream.net`:
```txt theme={null}
myemailaddr+test@pipedream.net
myemailaddr+unsubscribe@pipedream.net
```
This allows you implement conditional logic in your workflow based on the data in that string.
### Troubleshooting
#### I’m receiving an `Expired Token` error when trying to read an email attachment
Email attachments are saved to S3, and are accessible in your workflows over [pre-signed URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ShareObjectPreSignedURL.html).
If the presigned URL for the attachment has expired, then you’ll need to send another email to create a brand new pre-signed URL.
If you’re using email attachments in combination with [`$.flow.delay`](/docs/workflows/building-workflows/code/nodejs/delay/) or [`$.flow.rerun`](/docs/workflows/building-workflows/code/nodejs/rerun/) which introduces a gap of time between steps in your workflow, then there’s a chance the email attachment’s URL will expire.
To overcome this, we suggest uploading your email attachments to your Project’s [File Store](/docs/workflows/data-management/file-stores/) for persistent storage.
## RSS
Choose the RSS trigger to watch an RSS feed for new items:
![]()
This will create an RSS [event source](/docs/workflows/building-workflows/triggers/) that polls the feed for new items on the schedule you select. Every time a new item is found, your workflow will run.
## Events
Events trigger workflow executions. The event that triggers your workflow depends on the trigger you select for your workflow:
* [HTTP triggers](/docs/workflows/building-workflows/triggers/#http) invoke your workflow on HTTP requests.
* [Cron triggers](/docs/workflows/building-workflows/triggers/#schedule) invoke your workflow on a time schedule (e.g., on an interval).
* [Email triggers](/docs/workflows/building-workflows/triggers/#email) invoke your workflow on inbound emails.
* [Event sources](/docs/workflows/building-workflows/triggers/#app-based-triggers) invoke your workflow on events from apps like Twitter, Google Calendar, and more.
### Selecting a test event
When you test any step in your workflow, Pipedream passes the test event you select in the trigger step:
![]()
You can select any event you’ve previously sent to your trigger as your test event, or send a new one.
### Examining event data
When you select an event, you’ll see [the incoming event data](/docs/workflows/building-workflows/triggers/#event-format) and the [event context](/docs/workflows/building-workflows/triggers/#stepstriggercontext) for that event:
![]()
Pipedream parses your incoming data and exposes it in the variable [`steps.trigger.event`](/docs/workflows/building-workflows/triggers/#event-format), which you can access in any [workflow step](/docs/workflows/#steps).
### Copying references to event data
When you’re [examining event data](/docs/workflows/building-workflows/triggers/#examining-event-data), you’ll commonly want to copy the name of the variable that points to the data you need to reference in another step.
Hover over the property whose data you want to reference, and click the **Copy Path** button to its right:
![]()
### Copying the values of event data
You can also copy the value of specific properties of your event data. Hover over the property whose data you want to copy, and click the **Copy Value** button to its right:
![]()
### Event format
When you send an event to your workflow, Pipedream takes the trigger data — for example, the HTTP payload, headers, etc. — and adds our own Pipedream metadata to it.
**This data is exposed in the `steps.trigger.event` variable. You can reference this variable in any step of your workflow**.
You can reference your event data in any [code](/docs/workflows/building-workflows/code/) or [action](/docs/components/contributing/#actions) step. See those docs or the general [docs on passing data between steps](/docs/workflows/#steps) for more information.
The specific shape of `steps.trigger.event` depends on the trigger type:
#### HTTP
| Property | Description |
| ----------- | ----------------------------------------------------- |
| `body` | A string or object representation of the HTTP payload |
| `client_ip` | IP address of the client that made the request |
| `headers` | HTTP headers, represented as an object |
| `method` | HTTP method |
| `path` | HTTP request path |
| `query` | Query string |
| `url` | Request host + path |
#### Cron Scheduler
| Property | Description |
| --------------------- | ----------------------------------------------------------------------------------------------- |
| `interval_seconds` | The number of seconds between scheduled executions |
| `cron` | When you’ve configured a custom cron schedule, the cron string |
| `timestamp` | The epoch timestamp when the workflow ran |
| `timezone_configured` | An object with formatted datetime data for the given execution, tied to the schedule’s timezone |
| `timezone_utc` | An object with formatted datetime data for the given execution, tied to the UTC timezone |
#### Email
We use Amazon SES to receive emails for the email trigger. You can find the shape of the event in the [SES docs](https://docs.aws.amazon.com/ses/latest/DeveloperGuide/receiving-email-notifications-contents.html).
### `steps.trigger.context`
`steps.trigger.event` contain your event’s **data**. `steps.trigger.context` contains *metadata* about the workflow and the execution tied to this event.
You can use the data in `steps.trigger.context` to uniquely identify the Pipedream event ID, the timestamp at which the event invoked the workflow, and more:
| Property | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `deployment_id` | A globally-unique string representing the current version of the workflow |
| `emitter_id` | The ID of the workflow trigger that emitted this event, e.g. the [event source](/docs/workflows/building-workflows/triggers/) ID. |
| `id` | A unique, Pipedream-provided identifier for the event that triggered this workflow |
| `owner_id` | The Pipedream-assigned [workspace ID](/docs/workspaces/#finding-your-workspaces-id) that owns the workflow |
| `platform_version` | The version of the Pipedream execution environment this event ran on |
| `replay` | A boolean, whether the event was replayed via the UI |
| `trace_id` | Holds the same value for all executions tied to an original event. [See below for more details](/docs/workflows/building-workflows/triggers/#how-do-i-retrieve-the-execution-id-for-a-workflow). |
| `ts` | The ISO 8601 timestamp at which the event invoked the workflow |
| `workflow_id` | The workflow ID |
| `workflow_name` | The workflow name |
#### How do I retrieve the execution ID for a workflow?
Pipedream exposes two identifies for workflow executions: one for the execution, and one for the “trace”.
`steps.trigger.context.id` should be unique for every execution of a workflow.
`steps.trigger.context.trace_id` will hold the same value for all executions tied to the same original event, e.g. if you have auto-retry enabled and it retries a workflow three times, the `id` will change, but the `trace_id` will remain the same. For example, if you call `$.flow.suspend()` on a workflow, we run a new execution after the suspend, so you’d see two total executions: `id` will be unique before and after the suspend, but `trace_id` will be the same.
You may notice other properties in `context`. These are used internally by Pipedream, and are subject to change.
### Event retention
On the Free and Basic plans, each workflow retains at most 100 events or 7 days of history.
* After 100 events have been processed, Pipedream will delete the oldest event data as new events arrive, keeping only the last 100 events.
* Or if an event is older than 7 days, Pipedream will delete the event data.
Other paid plans have longer retention. [See the pricing page](https://pipedream.com/pricing) for details.
Events are also stored in [event history](/docs/workflows/event-history/) for up to 30 days, depending on your plan. [See the pricing page](https://pipedream.com/pricing) for the retention on your plan.
Events that are [delayed](/docs/workflows/building-workflows/control-flow/delay/) or [suspended](/docs/glossary/#suspend) are retained for the duration of the delay. After the delay, the workflow is executed, and the event data is retained according to the rules above.
For an extended history of events across all of your workflows, included processed events, with the ability to filter by status and time range, please see the [Event History](/docs/workflows/event-history/).
## Don’t see a trigger you need?
If you don’t see a trigger you’d like us to support, please [let us know](https://pipedream.com/support/).
# Using Props
Source: https://pipedream.com/docs/workflows/building-workflows/using-props
Props are fields that can be added to code steps in a workflow to abstract data from the code and improve reusability. Most actions use props to capture user input (e.g., to allow users to customize the URL, method and payload for the Send HTTP Request action). Props support the entry of simple values (e.g., `hello world` or `123`) or expressions in `{{ }}` that can reference objects in scope or run basic Node.js code.
## Entering Expressions
Expressions make it easy to pass data exported from previous steps into a code step or action via props. For example, if your workflow is triggered on new Tweets and you want to send the Tweet content to an HTTP or webhook destination, you would reference `{{steps.trigger.event.body}}` to do that.
While the data expected by each input depends on the data type (e.g., string, integer, array, etc) and the data entry mode (structured or non-structured — if applicable), the format for entering expressions is always the same; expressions are always enclosed in `{{ }}`.
There are three ways to enter expressions in a prop field — you can use the object explorer, enter it manually, or paste a reference from a step export.
### Use the object explorer
When you click into a prop field, an object explorer expands below the input. You can explore all the objects in scope, filter for keywords (e.g., a key name), and then select the element to insert into the form as an expression.
![]()
### Manually enter or edit an expression
To manually enter or edit an expression, just enter or edit a value between double curly braces `{{ }}`. Pipedream provides auto-complete support as soon as you type.
![]()
You can also run Node.js code in `{{ }}`. For example, if `event.foo` is a JSON object and you want to pass it to a param as a string, you can run `{{JSON.stringify(event.foo)}}`.
### Paste a reference from a step export
To paste a reference from a step export, find the reference you want to use, click **Copy Path** and then paste it into the input.
![]()
# Data Stores
Source: https://pipedream.com/docs/workflows/data-management/data-stores
**Data stores** are Pipedream’s built-in key-value store.
Data stores are useful for:
* Storing and retrieving data at a specific key
* Setting automatic expiration times for temporary data (TTL)
* Counting or summing values over time
* Retrieving JSON-serializable data across workflow executions
* Caching and rate limiting
* And any other case where you’d use a key-value store
You can connect to the same data store across workflows, so they’re also great for sharing state across different services.
You can use pre-built, no-code actions to store, update, and clear data, or interact with data stores programmatically in [Node.js](/docs/workflows/building-workflows/code/nodejs/using-data-stores/) or [Python](/docs/workflows/building-workflows/code/python/using-data-stores/).
## Using pre-built Data Store actions
Pipedream provides several pre-built actions to set, get, delete, and perform other operations with data stores.
### Inserting data
To insert data into a data store:
1. Add a new step to your workflow.
2. Search for the **Data Stores** app and select it.
3. Select the **Add or update a single record** pre-built action.
![]()
Configure the action:
1. **Select or create a Data Store** — create a new data store or choose an existing data store.
2. **Key** - the unique ID for this data that you’ll use for lookup later
3. **Value** - The data to store at the specified `key`
4. **Time to Live (TTL)** - (Optional) The number of seconds until this record expires and is automatically deleted. Leave blank for records that should not expire.
![]()
For example, to store the timestamp when the workflow was initially triggered, set the **Key** to **Triggered At** and the **Value** to `{{steps.trigger.context.ts}}`.
The **Key** must evaluate to a string. You can pass a static string, reference [exports](/docs/workflows/#step-exports) from a previous step, or use [any valid expression](/docs/workflows/building-workflows/using-props/#entering-expressions).
![]()
Need to store multiple records in one action? Use the **Add or update multiple records** action instead.
### Retrieving Data
The **Get record** action will retrieve the latest value of a data point in one of your data stores.
1. Add a new step to your workflow.
2. Search for the **Data Stores** app and select it.
3. Select the **Add or update a single record** pre-built action.
![]()
Configure the action:
1. **Select or create a Data Store** — create a new data store or choose an existing data store.
2. **Key** - the unique ID for this data that you’ll use for lookup later
3. **Create new record if key is not found** - if the specified key isn’t found, you can create a new record
4. **Value** - The data to store at the specified `key`
![]()
### Setting or updating record expiration (TTL)
You can set automatic expiration times for records using the **Update record expiration** action:
1. Add a new step to your workflow.
2. Search for the **Data Stores** app and select it.
3. Select the **Update record expiration** pre-built action.
Configure the action:
1. **Select a Data Store** - select the data store containing the record to modify
2. **Key** - the key for the record you want to update the expiration for
3. **Expiration Type** - choose from preset expiration times (1 hour, 1 day, 1 week, etc.) or select “Custom value” to enter a specific time in seconds
4. **Custom TTL (seconds)** - (only if “Custom value” is selected) enter the number of seconds until the record expires
To remove expiration from a record, select “No expiration” as the expiration type.
### Deleting Data
To delete a single record from your data store, use the **Delete a single record** action in a step:
![]()
Then configure the action:
1. **Select a Data Store** - select the data store that contains the record to be deleted
2. **Key** - the key that identifies the individual record
For example, you can delete the data at the **Triggered At** key that we’ve created in the steps above:
![]()
Deleting a record does not delete the entire data store. [To delete an entire data store, use the Pipedream Data Stores Dashboard](/docs/workflows/data-management/data-stores/#deleting-data-stores).
## Managing data stores
You can view the contents of your data stores at any time in the [Pipedream Data Stores dashboard](https://pipedream.com/data-stores/). You can also add, edit, or delete data store records manually from this view.
### Editing data store values manually
1. Select the data store
2. Click the pencil icon on the far right of the record you want to edit. This will open a text box that will allow you to edit the contents of the value. When you’re finished with your edits, save by clicking the checkmark icon.
![]()
### Deleting data stores
You can delete a data store from this dashboard as well. On the far right in the data store row, click the trash can icon.
![]()
**Deleting a data store is irreversible**.
If the **Delete** option is greyed out and unclickable, you have workflows using the data store in a step. Click the **>** to the left of the data store’s name to expand the linked workflows.
![]()
Then remove the data store from any linked steps.
## Using data stores in code steps
Refer to the [Node.js](/docs/workflows/building-workflows/code/nodejs/using-data-stores/) and [Python](/docs/workflows/building-workflows/code/python/using-data-stores/) data store docs to learn how to use data stores in code steps. You can get, set, delete and perform any other data store operations in code. You cannot use data stores in [Bash](/docs/workflows/building-workflows/code/bash/) or [Go](/docs/workflows/building-workflows/code/go/) code steps.
## Compression
Data saved in data stores is [Brotli-compressed](https://github.com/google/brotli), minimizing storage. The total compression ratio depends on the data being compressed. To test this on your own data, run it through a package that supports Brotli compression and measure the size of the data before and after.
## Data store limits
Depending on your plan, Pipedream sets limits on:
1. The total number of data stores
2. The total number of keys across all data stores
3. The total storage used across all data stores, [after compression](/docs/workflows/data-management/data-stores/#compression)
You’ll find your workspace’s limits in the **Data Stores** section of usage dashboard in the bottom-left of [https://pipedream.com](https://pipedream.com).
![]()
## Atomic operations
Data store operations are not atomic or transactional, which can lead to race conditions. To ensure atomic operations, be sure to limit access to a data store key to a [single workflow with a single worker](/docs/workflows/building-workflows/settings/concurrency-and-throttling/) or use a service that supports atomic operations from among our [integrated apps](https://pipedream.com/apps).
## Supported data types
Data stores can hold any JSON-serializable data within the storage limits. This includes data types including:
* Strings
* Objects
* Arrays
* Dates
* Integers
* Floats
But you cannot serialize functions, classes, sets, maps, or other complex objects.
## Exporting data to an external service
In order to stay within the [data store limits](/docs/workflows/data-management/data-stores/#data-store-limits), you may need to export the data in your data store to an external service.
The following Node.js example action will export the data in chunks via an HTTP POST request. You may need to adapt the code to your needs. Click on [this link](https://pipedream.com/new?h=tch_egfAMv) to create a copy of the workflow in your workspace.
If the data contained in each key is large, consider lowering the number of `chunkSize`.
* Adjust your [workflow memory and timeout settings](/docs/workflows/building-workflows/settings/) according to the size of the data in your data store. Set the memory at 512 MB and timeout to 60 seconds and adjust higher if needed.
* Monitor the exports of this step after each execution for any potential errors preventing a full export. Run the step as many times as needed until all your data is exported.
This action deletes the keys that were successfully exported. It is advisable to first run a test without deleting the keys. In case of any unforeseen errors, your data will still be safe.
```javascript theme={null}
import { axios } from "@pipedream/platform";
export default defineComponent({
props: {
dataStore: {
type: "data_store",
},
chunkSize: {
type: "integer",
label: "Chunk Size",
description: "The number of items to export in one request",
default: 100,
},
shouldDeleteKeys: {
type: "boolean",
label: "Delete keys after export",
description: "Whether the data store keys will be deleted after export",
default: true,
},
},
methods: {
async *chunkAsyncIterator(asyncIterator, chunkSize) {
let chunk = [];
for await (const item of asyncIterator) {
chunk.push(item);
if (chunk.length === chunkSize) {
yield chunk;
chunk = [];
}
}
if (chunk.length > 0) {
yield chunk;
}
},
},
async run({ steps, $ }) {
const iterator = this.chunkAsyncIterator(this.dataStore, this.chunkSize);
for await (const chunk of iterator) {
try {
// export data to external service
await axios($, {
url: "https://external_service.com",
method: "POST",
data: chunk,
// may need to add authentication
});
// delete exported keys and values
if (this.shouldDeleteKeys) {
await Promise.all(chunk.map(([key]) => this.dataStore.delete(key)));
}
console.log(
`number of remaining keys: ${(await this.dataStore.keys()).length}`
);
} catch (e) {
// an error occurred, don't delete keys
console.log(`error exporting data: ${e}`);
}
}
},
});
```
# Connecting to Databases
Source: https://pipedream.com/docs/workflows/data-management/databases
Connecting to a database is essential for developing production workflows. Whether you’re storing application data, querying user information, or analyzing event logs, most workflows and serverless functions require querying data at some point.
Pipedream workflows run in the AWS `us-east-1` network, sending requests from standard AWS IP ranges.
## Connecting to Restricted Databases
**Unless your database is publicly accessible, you’ll likely need to add specific IPs to its allow-list.** To do this, you can configure your database connection to use either a shared or dedicated static IP address from Pipedream:
### Create a Dedicated Static IP for Outbound Traffic
* [Virtual Private Clouds (VPCs)](/docs/workflows/vpc/) in Pipedream let you deploy any workflow to a private network and is the most secure and recommended approach to using a static IP.
* Once configured, the VPC will give you a dedicated egress IP that’s unique to your workspace, and is available to any workflow within your workspace.
### Send Requests from a Shared Static IP
* When configuring your database connection as a [connected account](/docs/apps/connected-accounts/) to Pipedream, you can choose to route network requests through a static IP block for [any app that’s supported by Pipedream’s SQL Proxy](/docs/workflows/data-management/databases/#supported-databases)
* Pipedream’s SQL Proxy routes requests to your database from the IP block below.
#### Supported Databases
Pipedream’s SQL Proxy, which enables the shared static IP, currently supports [MySQL](https://pipedream.com/apps/mysql), [PostgreSQL](https://pipedream.com/apps/postgresql), and [Snowflake](https://pipedream.com/apps/snowflake). Please let us know if you’d like to see support for other database types.
#### Enabling the Shared Static IP
Connect your account for one of the [supported database apps](/docs/workflows/data-management/databases/#supported-databases) and set **Use Shared Static IP** to **TRUE**, then click **Test connection** to ensure Pipedream can successfully connect to your database.
![]()
#### Shared Static IP Block
Add the following IP block to your database allow-list:
```
44.223.89.56/29
```
## FAQ
### What’s the difference between using a shared static IP with the SQL Proxy vs a dedicated IP using a VPC?
Both the SQL Proxy and VPCs enable secure database connections from a static IP.
* VPCs offer enhanced isolation and security by providing a **dedicated** static IP for workflows within your workspace
* The SQL proxy routes requests to your database connections through a set of **shared** static IPs
# Working with SQL
Source: https://pipedream.com/docs/workflows/data-management/databases/working-with-sql
Pipedream makes it easy to interact with SQL databases within your workflows. You can securely connect to your database and use either pre-built no-code triggers and actions to interact with your database, or execute custom SQL queries.
## SQL Editor
With the built-in SQL editor, you access linting and auto-complete features typical of modern SQL editors.
![]()
## Schema Explorer
When querying a database, you need to understand the schema of the tables you’re working with. The schema explorer provides a visual interface to explore the tables in your database, view their columns, and understand the relationships between them.
* Once you connect your account with one of the [supported database apps](/docs/workflows/data-management/databases/working-with-sql/#supported-databases), we automatically fetch and display the details of the database schema below
* You can **view the columns of a table**, their data types, and relationships between tables
* You can also **search and filter** the set of tables that are listed in your schema based on table or column name
![]()
## Prepared Statements
Prepared statements let you safely execute SQL queries with dynamic inputs that are automatically defined as parameters, in order to help prevent SQL injection attacks.
To reference dynamic data in a SQL query, simply use the standard `{{ }}` notation just like any other code step in Pipedream. For example,
```sql theme={null}
select *
from products
where name = {{steps.get_product_info.$return_value.name}}
and created_at > {{steps.get_product_info.$return_value.created_at}}
```
**Prepared statement:**
![]()
**Computed statement:**
![]()
When you include step references in your SQL query, Pipedream automatically converts your query to a prepared statement using placeholders with an array of params.
Below your query input, you can toggle between the computed and prepared statements.
## Getting Started
* From the step selector in the builder, select the **Query a Database** action for the [relevant database app](/docs/workflows/data-management/databases/working-with-sql/#supported-databases)
![]()
* If you already have a connected account, select it from the dropdown. Otherwise, click **Connect Account** to configure the database connection.
* Follow the prompts to connect your account, then click **Test connection** to ensure Pipedream can successfully connect to your database
![]()
* Once you’ve successfully connected your account, you can explore the [database schema](/docs/workflows/data-management/databases/working-with-sql/#schema-explorer) to understand the tables and columns in your database
* Write your SQL query in the editor — read more about [prepared statements](/docs/workflows/data-management/databases/working-with-sql/#prepared-statements) above to reference dynamic data in your query
### Supported Databases
The [SQL editor](/docs/workflows/data-management/databases/working-with-sql/#sql-editor), [schema explorer](/docs/workflows/data-management/databases/working-with-sql/#schema-explorer), and support for [prepared statements](/docs/workflows/data-management/databases/working-with-sql/#prepared-statements) are currently supported for these database apps:
* [MySQL](https://pipedream.com/apps/mysql)
* [PostgreSQL](https://pipedream.com/apps/postgresql)
* [Snowflake](https://pipedream.com/apps/snowflake)
Need to query a different database type? Let us know!
# Destinations
Source: https://pipedream.com/docs/workflows/data-management/destinations
**Destinations**, like [actions](/docs/components/contributing/#actions), abstract the delivery and connection logic required to send events to services like Amazon S3, or targets like HTTP and email.
However, Destinations are different than actions in two ways:
* Events are delivered to the Destinations asynchronously, after your workflow completes. This means you don’t wait for network I/O (e.g. for HTTP requests or connection overhead for data warehouses) within your workflow code, so you can process more events faster.
* In the case of data stores like S3, you typically don’t want to send every event on its own. This can be costly and carries little benefit. Instead, you typically want to batch a collection of events together, sending the batch at some frequency. Destinations handle that batching for relevant services.
The docs below discuss features common to all Destinations. See the [docs for a given destination](/docs/workflows/data-management/destinations/#available-destinations) for information specific to those destinations.
## Available Destinations
* [HTTP](/docs/workflows/data-management/destinations/http/)
* [Email](/docs/workflows/data-management/destinations/email/)
* [S3](/docs/workflows/data-management/destinations/s3/)
* [SSE](/docs/workflows/data-management/destinations/sse/)
* [Emit to another listener](/docs/workflows/data-management/destinations/emit/)
## Using destinations
### Using destinations in workflows
You can send data to Destinations in [Node.js code steps](/docs/workflows/building-workflows/code/nodejs/), too, using `$.send` functions.
`$.send` is an object provided by Pipedream that exposes destination-specific functions like `$.send.http()`, `$.send.s3()`, and more. This allows you to send data to destinations programmatically, if you need more control than the default actions provide.
Let’s use `$.send.http()` to send an HTTP POST request like we did in the Action example above. [Add a new action](/docs/workflows/building-workflows/actions/), then search for “**Run custom code**”:
Create a new HTTP endpoint URL (try creating a new Pipedream workflow and adding an HTTP trigger), and add the code below to your code step, with the URL you created:
```javascript theme={null}
export default defineComponent({
async run({ steps, $}) {
$.send.http({
method: "POST",
url: "[YOUR URL HERE]",
data: {
name: "Luke Skywalker",
},
});
}
})
```
See the docs for the [HTTP destination](/docs/workflows/data-management/destinations/http/) to learn more about all the options you can pass to the `$.send.http()` function.
Again, it’s important to remember that **Destination delivery is asynchronous**. If you iterate over an array of values and send an HTTP request for each:
```javascript theme={null}
export default defineComponent({
async run({ steps, $}) {
const names = ["Luke", "Han", "Leia", "Obi Wan"];
for (const name of names) {
$.send.http({
method: "POST",
url: "[YOUR URL HERE]",
data: {
name,
},
});
}
}
})
```
you won’t have to `await` the execution of the HTTP requests in your workflow. We’ll collect every `$.send.http()` call and defer those HTTP requests, sending them after your workflow finishes.
### Using destinations in actions
If you’re authoring a [component action](/docs/components/contributing/#actions), you can deliver data to destinations, too. `$.send` isn’t directly available to actions like it is for workflow code steps. Instead, you use `$.send` to access the destination-specific functions:
```javascript theme={null}
export default {
name: "Action Demo",
key: "action_demo",
version: "0.0.1",
type: "action",
async run({ $ }) {
$.send.http({
method: "POST",
url: "[YOUR URL HERE]",
data: {
name: "Luke Skywalker",
},
});
}
}
```
[See the component action API docs](/docs/components/contributing/api/#actions) for more details.
## Asynchronous Delivery
Events are delivered to destinations *asynchronously* — that is, separate from the execution of your workflow. **This means you’re not waiting for network or connection I/O in the middle of your function, which can be costly**.
Some destination payloads, like HTTP, are delivered within seconds. For other destinations, like S3 and SQL, we collect individual events into a batch and send the batch to the destination. See the [docs for a specific destination](/docs/workflows/data-management/destinations/#available-destinations) for the relevant batch delivery frequency.
# Email
Source: https://pipedream.com/docs/workflows/data-management/destinations/email
The Email Destination allows you send an email to *yourself* — the email address tied to the account you signed up with — at any step of a workflow.
You can use this to email yourself when you receive a specific event, for example when a user signs up on your app. You can send yourself an email when a cron job finishes running, or when a job fails. Anywhere you need an email notification, you can use the Email Destination!
## Adding an Email Destination
### Adding an Email Action
1. Add a new step to your workflow
2. Select the **Send Yourself an Email** Action. You can modify the **Subject** and the message (either **Plain Text** or **HTML**) however you want.
### Using `$.send.email` in workflows
You can send data to an Email Destination in [Node.js code steps](/docs/workflows/building-workflows/code/nodejs/), too, using the `$.send.email()` function. **This allows you to send emails to yourself programmatically, if you need more control than actions provide**.
`$.send.email()` takes the same parameters as the corresponding action:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.email({
subject: "Your subject",
text: "Plain text email body",
html: "HTML email body"
});
}
});
```
The `html` property is optional. If you include both the `text` and `html` properties, email clients that support HTML will prefer that over the plaintext version.
Like with any `$.send` function, you can use `$.send.email()` conditionally, within a loop, or anywhere you’d use a function normally in Node.js.
### Using `$.send.email` in component actions
If you’re authoring a [component action](/docs/components/contributing/#actions), you can deliver data to an email destination using `$.send.email`.
`$.send.email` functions the same as [`$.send.email` in workflow code steps](/docs/workflows/data-management/destinations/email/#using-sendemail-in-workflows):
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.email({
subject: "Your subject",
text: "Plain text email body",
html: "HTML email body"
});
}
})
```
## Delivery details
All emails come from **[notifications@pipedream.com](mailto:notifications@pipedream.com)**.
# Emit Events
Source: https://pipedream.com/docs/workflows/data-management/destinations/emit
Like [event sources](/docs/workflows/building-workflows/triggers/), workflows can emit events. These events can trigger other workflows, or be consumed using Pipedream’s [REST API](/docs/rest-api/#get-workflow-emits).
## Using `$.send.emit()` in workflows
You can emit arbitrary events from any [Node.js code steps](/docs/workflows/building-workflows/code/nodejs/) using `$.send.emit()`.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.emit({
name: "Yoda",
});
}
});
```
`$.send.emit()` accepts an object with the following properties:
```javascript theme={null}
$.send.emit(
event, // An object that contains the event you'd like to emit
channel, // Optional, a string specifying the channel
);
```
## Emitting events to channels
By default, events are emitted to the default channel. You can optionally emit events to a different channel, and listening sources or workflows can subscribe to events on this channel, running the source or workflow only on events emitted to that channel.
Pass the channel as the second argument to `$.send.emit()`:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.emit(
{
name: "Yoda",
},
'channel_name'
);
}
});
```
## Using `$.send.emit()` in component actions
If you’re authoring a [component action](/docs/components/contributing/#actions), you can emit data using `$.send.emit()`.
`$.send.emit()` functions the same as [`$.send.emit()` in workflow code steps](/docs/workflows/data-management/destinations/emit/#using-sendemit-in-workflows):
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.emit({
name: "Yoda",
});
}
})
```
**Destination delivery is asynchronous**: emits are sent after your workflow finishes.
You can call `$.send.emit()` multiple times within a workflow, for example: to iterate over an array of values and emit an event for each.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
const names = ["Luke", "Han", "Leia", "Obi Wan"];
for (const name of names) {
$.send.emit({
name,
});
}
}
});
```
## Trigger a workflow from emitted events
We call the events you emit from a workflow **emitted events**. Sometimes, you’ll want emitted events to trigger another workflow. This can be helpful when:
* You process events from different workflows in the same way. For example, you want to log events from many workflows to Amazon S3 or a logging service. You can write one workflow that handles logging, then `$.send.emit()` events from other workflows that are consumed by the single, logging workflow. This helps remove duplicate logic from the other workflows.
* Your workflow is complex and you want to separate it into multiple workflows to group logical functions together. You can `$.send.emit()` events from one workflow to another to chain the workflows together.
Here’s how to configure a workflow to listen for emitted events.
1. Currently, you can’t select emitted events as a workflow trigger from the Pipedream UI. We’ll show you how add the trigger via API. First, pick an existing workflow where you’d like to receive emitted events. **If you want to start with a [new workflow](https://pipedream.com/new), just select the HTTP / Webhook trigger**.
2. This workflow is called the **listener**. The workflow where you’ll use `$.send.emit()` is called the **emitter**. If you haven’t created the emitter workflow yet, [do that now](https://pipedream.com/new).
3. Get the workflow IDs of both the listener and emitter workflows. **You’ll find the workflow ID in the workflow’s URL in your browser bar —it’s the `p_abc123` in `https://pipedream.com/@username/p_abc123/`**.
4. You can use the Pipedream REST API to configure the listener to receive events from the emitter. We call this [creating a subscription](/docs/rest-api/#listen-for-events-from-another-source-or-workflow). If your listener’s ID is `p_abc123` and your emitter’s ID is `p_def456`, you can run the following command to create this subscription:
```bash theme={null}
curl "https://api.pipedream.com/v1/subscriptions?emitter_id=dc_def456&listener_id=p_abc123" \
-X POST \
-H "Authorization: Bearer " \
-H "Content-Type: application/json"
```
5. Run your emitter workflow, emitting an event using `$.send.emit()`:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.emit({
name: "Yoda",
});
}
});
```
This should trigger your listener, and you should see the same event in [the event inspector](/docs/workflows/building-workflows/inspect/#the-inspector).
**Note**: Please upvote [this issue](https://github.com/PipedreamHQ/pipedream/issues/682) to see support for *adding* emitted events as a workflow trigger in the UI.
## Consuming emitted events via REST API
`$.send.emit()` can emit any data you’d like. You can retrieve that data using Pipedream’s REST API endpoint for [retrieving emitted events](/docs/rest-api/#get-workflow-emits).
This can be helpful when you want a workflow to process data asynchronously using a workflow. You can save the results of your workflow with `$.send.emit()`, and only retrieve the results in batch when you need to using the REST API.
## Emit logs / troubleshooting
Below your code step, you’ll see both the data that was sent in the emit. If you ran `$.send.emit()` multiple times within the same code step, you’ll see the data that was emitted for each.
# HTTP
Source: https://pipedream.com/docs/workflows/data-management/destinations/http
HTTP Destinations allow you to send data to another HTTP endpoint URL outside of Pipedream. This can be an endpoint you own and operate, or a URL tied to a service you use (for example, a [Slack Incoming Webhook](https://api.slack.com/incoming-webhooks)).
## Using `$.send.http` in workflows
You can send HTTP requests in [Node.js code steps](/docs/workflows/building-workflows/code/nodejs/) using `$.send.http()`.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.http({
method: "POST",
url: "[YOUR URL HERE]",
data: {
name: "Luke Skywalker",
},
});
}
});
```
`$.send.http()` accepts an object with all of the following properties:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.http({
method, // Required, HTTP method, a string, e.g. POST, GET
url, // Required, the URL to send the HTTP request to
data, // HTTP payload
headers, // An object containing custom headers, e.g. { "Content-Type": "application/json" }
params, // An object containing query string parameters as key-value pairs
auth, // An object that contains a username and password property, for HTTP basic auth
});
}
});
```
**Destination delivery is asynchronous**: the HTTP requests are sent after your workflow finishes. This means **you cannot write code that operates on the HTTP response**. These HTTP requests **do not** count against your workflow’s compute time.
If you iterate over an array of values and send an HTTP request for each:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
const names = ["Luke", "Han", "Leia", "Obi Wan"];
names.forEach((name) => {
$.send.http({
method: "POST",
url: "[YOUR URL HERE]",
data: {
name,
},
});
});
}
});
```
you won’t have to `await` the execution of the HTTP requests in your workflow. We’ll collect every `$.send.http()` call and defer those HTTP requests, sending them after your workflow finishes.
## Using `$.send.http` in component actions
If you’re authoring a [component action](/docs/components/contributing/#actions), you can deliver data to an HTTP destination using `$.send.http`.
`$.send.http` functions the same as [`$.send.http` in workflow code steps](/docs/workflows/data-management/destinations/http/#using-sendhttp-in-workflows):
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.http({
method: "GET",
url: "https://example.com"
})
}
});
```
## HTTP Destination delivery
HTTP Destination delivery is handled asynchronously, separate from the execution of a workflow. However, we deliver the specified payload to HTTP destinations for every event sent to your workflow.
Generally, this means it should only take a few seconds for us to send the event to the destination you specify. In some cases, delivery will take longer.
The time it takes to make HTTP requests sent with `$.send.http()` does not count against your workflow quota.
## HTTP request and response logs
Below your code step, you’ll see both the data that was sent in the HTTP request, and the HTTP response that was issued. If you issue multiple HTTP requests, we’ll show the request and response data for each.
## What if I need to access the HTTP response in my workflow?
Since HTTP requests sent with `$.send.http()` are sent asynchronously, after your workflow runs, **you cannot access the HTTP response in your workflow**.
If you need to access the HTTP response data in your workflow, [use `axios`](/docs/workflows/building-workflows/code/nodejs/http-requests/) or another HTTP client.
## Timeout
The timeout on HTTP request sent with `$.send.http()` is currently **5 seconds**. This time includes DNS resolution, connecting to the host, writing the request body, server processing, and reading the response body.
Any requests that exceed 5 seconds will yield a `timeout` error.
## Retries
Currently, Pipedream will not retry any failed request. If your HTTP destination endpoint is down, or returns an error response, we’ll display that response in the observability associated with the Destination in the relevant step.
## IP addresses for Pipedream HTTP requests
These IP addresses are tied to **requests sent with `$.send.http` only, not other HTTP requests made from workflows**. To whitelist standard HTTP requests from Pipedream workflows, [use VPCs](/docs/workflows/vpc/).
When you make an HTTP request using `$.send.http()`, the traffic will come from one of the following IP addresses:
```
3.208.254.1053.212.246.1733.223.179.1313.227.157.1893.232.105.553.234.187.12618.235.13.18234.225.84.3152.2.233.852.23.40.20852.202.86.952.207.145.19054.86.100.5054.88.18.8154.161.28.250107.22.76.172
```
This list may change over time. If you’ve previously whitelisted these IP addresses and are having trouble sending HTTP requests to your target service, please check to ensure this list matches your firewall rules.
# Amazon S3
Source: https://pipedream.com/docs/workflows/data-management/destinations/s3
[Amazon S3](https://aws.amazon.com/s3/) — the Simple Storage Service — is a common place to dump data for long-term storage on AWS. Pipedream supports delivery to S3 as a first-class Destination.
## Using `$.send.s3` in workflows
You can send data to an S3 Destination in [Node.js code steps](/docs/workflows/building-workflows/code/nodejs/) using `$.send.s3()`.
`$.send.s3()` takes the following parameters:
```javascript theme={null}
$.send.s3({
bucket: "your-bucket-here",
prefix: "your-prefix/",
payload: event.body,
});
```
Like with any `$.send` function, you can use `$.send.s3()` conditionally, within a loop, or anywhere you’d use a function normally.
## Using `$.send.s3` in component actions
If you’re authoring a [component action](/docs/components/contributing/#actions), you can deliver data to an S3 destination using `$.send.s3`.
`$.send.s3` functions the same as [`$.send.s3` in workflow code steps](/docs/workflows/data-management/destinations/s3/#using-sends3-in-workflows):
```javascript theme={null}
async run({ $ }) {
$.send.s3({
bucket: "your-bucket-here",
prefix: "your-prefix/",
payload: event.body,
});
}
```
## S3 Bucket Policy
In order for us to deliver objects to your S3 bucket, you need to modify its [bucket policy](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/add-bucket-policy.html) to allow Pipedream to upload objects.
**Replace `[your bucket name]` with the name of your bucket** near the bottom of the policy.
```json theme={null}
{
"Version": "2012-10-17",
"Id": "allow-pipedream-limited-access",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::203863770927:role/Pipedream"
},
"Action": [
"s3:AbortMultipartUpload",
"s3:GetBucketLocation",
"s3:PutObject",
"s3:PutObjectAcl",
"s3:ListBucketMultipartUploads"
],
"Resource": [
"arn:aws:s3:::[your bucket name]",
"arn:aws:s3:::[your bucket name]/*"
]
}
]
}
```
This bucket policy provides the minimum set of permissions necessary for Pipedream to deliver objects to your bucket. We use the [Multipart Upload API](https://docs.aws.amazon.com/AmazonS3/latest/dev/uploadobjusingmpu.html) to upload objects, and require the [relevant permissions](https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuAndPermissions.html).
## S3 Destination delivery
S3 Destination delivery is handled asynchronously, separate from the execution of a workflow. **Moreover, events sent to an S3 bucket are batched and delivered once a minute**. For example, if you sent 30 events to an S3 Destination within a particular minute, we would collect all 30 events, delimit them with newlines, and write them to a single S3 object.
In some cases, delivery will take longer than a minute.
## S3 object format
We upload objects using the following format:
```txt theme={null}
[PREFIX]/YYYY/MM/DD/HH/YYYY-MM-DD-HH-MM-SS-IDENTIFIER.gz
```
That is — we write objects first to your prefix, then within folders specific to the current date and hour, then upload the object with the same date information in the object, so that it’s easy to tell when it was uploaded by object name alone.
For example, if I were writing data to a prefix of `test/`, I might see an object in S3 at this path:
```txt theme={null}
test/2019/05/25/16/2019-05-25-16-14-58-8f25b54462bf6eeac3ee8bde512b6c59654c454356e808167a01c43ebe4ee919.gz
```
As noted above, a given object contains all payloads delivered to an S3 Destination within a specific minute. Multiple events within a given object are newline-delimited.
## Limiting S3 Uploads by IP
S3 provides a mechanism to [limit operations only from specific IP addresses](https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3). If you’d like to apply that filter, uploads using `$.send.s3()` should come from one of the following IP addresses:
```txt theme={null}
3.208.254.1053.212.246.1733.223.179.1313.227.157.1893.232.105.553.234.187.12618.235.13.18234.225.84.3152.2.233.852.23.40.20852.202.86.952.207.145.19054.86.100.5054.88.18.8154.161.28.250107.22.76.172
```
This list may change over time. If you’ve previously whitelisted these IP addresses and are having trouble uploading S3 objects, please check to ensure this list matches your firewall rules.
# Server Sent Events (SSE)
Source: https://pipedream.com/docs/workflows/data-management/destinations/sse
Pipedream supports [Server-sent events](https://developer.mozilla.org/en-US/Web/API/Server-sent_events) (SSE) as a destination, enabling you to send events from a workflow directly to a client subscribed to the event stream.
## What is SSE?
[Server-sent Events](https://developer.mozilla.org/en-US/Web/API/Server-sent_events) (SSE) is a specification that allows servers to send events directly to clients that subscribe to those events, similar to [WebSockets](https://developer.mozilla.org/en-US/Web/API/WebSockets_API) and related server to client push technologies.
Unlike WebSockets, SSE enables one-way communication from server to clients (WebSockets enable bidirectional communication between server and client, allowing you to pass messages back and forth). Luckily, if you only need a client to subscribe to events from a server, and don’t require bidirectional communication, SSE is simple way to make that happen.
## What can I do with the SSE destination?
SSE is typically used by web developers to update a webpage with new events in real-time, without forcing a user to reload a page to fetch new data. If you’d like to update data on a webpage in that manner, you can subscribe to your workflow’s event stream and handle new events as they come in.
Beyond web browsers, any program that’s able to create an [`EventSource` interface](https://developer.mozilla.org/en-US/Web/API/EventSource) can listen for server-sent events delivered from Pipedream. You can run a Node.js script or a Ruby on Rails app that receives server-sent events, for example.
## Sending data to an SSE Destination in workflows
You can send data to an SSE Destination in [Node.js code steps](/docs/workflows/building-workflows/code/nodejs/) using the `$.send.sse()` function.
1. Add a new step to your workflow
2. Select the option to **Run custom code** and choose the Node.js runtime.
3. Add this code to that step:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.sse({
channel: "events", // Required, corresponds to the event in the SSE spec
payload: { // Required, the event payload
name: "Luke Skywalker"
}
});
}
});
```
**See [this workflow](https://pipedream.com/new?h=tch_mp7f6q)** for an example of how to use `$.send.sse()`.
Send a test event to your workflow, then review the section on [Receiving events](/docs/workflows/data-management/destinations/sse/#receiving-events) to see how you can setup an `EventSource` to retrieve events sent to the SSE Destination.
**Destination delivery is asynchronous**. If you iterate over an array of values and send an SSE for each:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
const names = ["Luke", "Han", "Leia", "Obi Wan"];
names.forEach(name => {
$.send.sse({
channel: "names",
payload: {
name
}
});
});
}
});
```
you won’t have to `await` the execution of the SSE Destination requests in your workflow. We’ll collect every `$.send.sse()` call and defer those requests, sending them after your workflow finishes.
## Using `$.send.sse` in component actions
If you’re authoring a [component action](/docs/components/contributing/#actions), you can send events to an SSE destination using `$.send.sse`.
`$.send.sse` functions the same as [`$.send.sse` in workflow code steps](/docs/workflows/data-management/destinations/sse/#sending-data-to-an-sse-destination-in-workflows):
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
$.send.sse({
channel: "events",
payload: {
name: "Luke Skywalker"
}
});
}
});
```
## Receiving events
Once you’ve sent events to an SSE Destination, you can start receiving a stream of those events in a client by configuring an [`EventSource`](https://developer.mozilla.org/en-US/Web/API/EventSource) that connects to the Pipedream SSE stream.
### Retrieving your workflow’s event stream URL
First, it’s important to note that all events sent to an SSE destination within a workflow are sent to an SSE event stream specific to that workflow. The event stream is tied to the workflow’s ID, which you can find by examining the URL of the pipeline in the Pipedream UI. For example, the `p_aBcDeF` in this URL is the pipeline ID:
![]()
**Note that the `p_` prefix is part of the workflow ID**.
Once you have the workflow ID, you can construct the event source URL for your SSE destination. That URL is of the following format:
```txt theme={null}
http://sdk.m.pipedream.net/pipelines/[YOUR WORKFLOW ID]/sse
```
In the example above, the URL of our event stream would be:
```txt theme={null}
http://sdk.m.pipedream.net/pipelines/p_aBcDeF/sse
```
You should be able to open that URL in your browser. Most modern browsers support connecting to an event stream directly, and will stream events without any work on your part to help you confirm that the stream is working.
If you’ve already sent events to your SSE destination, you should see those events here! We’ll return the most recent 100 events delivered to the corresponding SSE destination immediately. This allows your client to catch up with events previously sent to the destination. Then, any new events sent to the SSE destination while you’re connected will be delivered to the client.
### Sample code to connect to your event stream
It’s easy to setup a simple webpage to `console.log()` all events from an event stream. You can find a lot more examples of how to work with SSE on the web, but this should help you understand the basic concepts.
You’ll need to create two files in the same directory on your machine: an `index.html` file for the HTML.
**index.html**
```html theme={null}
SSE test
```
**Make sure to add your workflow ID and the name of your channel you specified in your SSE Destination**. Then, open the `index.html` page in your browser. In your browser’s developer tools JavaScript console, you should see new events appear as you send them.
Note that the `addEventListener` code will listen specifically for events sent to the **events** `channel` specified in our SSE destination. You can listen for multiple types of events at once by adding multiple event listeners on the client.
**Try triggering more test events from your workflow while this page is open to see how this works end-to-end**.
## `:keepalive` messages
[The SSE spec](https://www.w3.org/TR/2009/WD-eventsource-20090421/#notes) notes that
> Legacy proxy servers are known to, in certain cases, drop HTTP connections after a short timeout. To protect against such proxy servers, authors can include a comment line (one starting with a ’:’ character) every 15 seconds or so.
Roughly every 15 seconds, we’ll send a message with the `:keepalive` comment to keep open SSE connections alive. These comments should be ignored when you’re listening for messages using the `EventSource` interface.
# File Stores
Source: https://pipedream.com/docs/workflows/data-management/file-stores
In Preview
File Stores are available in Preview. There may be changes to allowed limits in the future.
If you have any feedback on File Stores, please let us know in our [community](https://pipedream.com/support).
File Stores are a filesystem that are scoped to a Project. All workflows within the same Project have access to the File Stores.
You can interact with these files through the Pipedream Dashboard or programmatically through your Project’s workflows.
Unlike files stored within a workflow’s `/tmp` directory which are subject to deletion between executions, File Stores are separate cloud storage. Files within a File Store can be long term storage accessible by your workflows.
![]()
## Managing File Stores from the Dashboard
You can access a File Store by opening the Project and selecting the *File Store* on the left hand navigation menu.
![]()
### Uploading files to the File Store
To upload a file, select *New* then select *File*:
![]()
Then in the new pop-up, you can either drag and drop or browser your computer to stage a file for uploading:
![]()
Now that the file(s) are staged for uploaded. Click *Upload* to upload them:
![]()
Finally, click *Done* to close the upload pop-up:
![]()
You should now see your file is uploaded and available for use within your Project:
![]()
### Deleting files from the File Store
You can delete individual files from a File Store by clicking the three dot menu on the far right of the file and selecting *Delete*.
![]()
After confirming that you want to delete the file, it will be permanently deleted.
File deletion is permanent
Once a file is deleted, it’s not possible to recover it. Please take care when deleting files from File Stores.
## Managing File Stores from Workflows
Files uploaded to a File Store are accessible by workflows within that same project.
You can access these files programmatically using the `$.files` helper within Node.js code steps.
File Stores are scoped to Projects
Only workflows within the same project as the File Store can access the files. Workflows outside of the project will not be able to access that project’s File Store.
### Listing files in the File Store
The `$.files.dir()` method allows you to list files and directories within the Project’s File Store. By default it will list the files at the root directory.
Here’s an example of how to iterate over the files in the root directory and open them as `File` instances:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// list all contents of the root File Stores directory in this project
const dirs = $.files.dir();
let files = [];
for await(const dir of dirs) {
// if this is a file, let's open it
if(dir.isFile()) {
files.push(dir.path)
}
}
return files
},
})
```
### Opening files
To interact with a file uploaded to the File Store, you’ll first need to open it.
Given there’s a file in the File Store called `example.png`, you can open it using the `$.files.open()` method:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Open the file by it's path in the File Store
const file = $.files.open('example.png')
// Log the S3 url to access the file publicly
return await file.toUrl()
},
})
```
Once the file has been opened, you can [read, write, delete the file and more](/docs/workflows/data-management/file-stores/reference/).
### Uploading files to File Stores
You can upload files using Node.js code in your workflows, either from URLs, from the `/tmp` directory in your workflows or directly from streams for high memory efficiency.
#### Uploading files from URLs
`File.fromUrl()` can upload a file from a public URL to the File Store.
First open a new file at a specific path in the File Store, and then pass a URL to the `fromUrl` method on that new file:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Upload a file to the File Store by a URL
const file = await $.files.open('pipedream.png').fromUrl('https://res.cloudinary.com/pipedreamin/image/upload/t_logo48x48/v1597038956/HzP2Yhq8_400x400_1_sqhs70.jpg')
// display the uploaded file's URL from the File Store:
console.log(await file.toUrl())
},
})
```
#### Uploading files from the workflow’s `/tmp` directory
`File.fromFile()` can upload a file stored within the workflow’s `/tmp` directory to the File Store.
First open a new file at a specific path in the File Store, and then pass a URL to the `fromFile` method on that new file:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Upload a file to the File Store from the local /tmp/ directory
const file = await $.files.open('recording.mp3').fromFile('/tmp/recording.mp3')
// Display the URL to the File Store hosted file
console.log(await file.toUrl())
},
})
```
#### Uploading files using streams
File Stores also support streaming to write large files. `File.createWriteStream()` creates a write stream for the file to upload to. Then you can pair this stream with a download stream from another remote location:
```javascript theme={null}
import { pipeline } from 'stream/promises';
import got from 'got'
export default defineComponent({
async run({ steps, $ }) {
const writeStream = await $.files.open('logo.png').createWriteStream()
const readStream = got.stream('https://pdrm.co/logo')
await pipeline(readStream, writeStream);
},
})
```
Additionally, you can pass a `ReadableStream` instance directly to a File instance:
```javascript theme={null}
import got from 'got'
export default defineComponent({
async run({ steps, $ }) {
// Start a new read stream
const readStream = got.stream('https://pdrm.co/logo')
// Populate the file's content from the read stream
await $.files.open("logo.png").fromReadableStream(readStream)
},
})
```
(Recommended) Pass the contentLength if possible
If possible, pass a `contentLength` argument, then File Store will be able to efficiently stream to use less memory. Without a `contentLength` argument, the entire file will need to be downloaded to `/tmp/` until it can be uploaded to the File store.
### Downloading files
File Stores live in cloud storage by default, but files can be downloaded to your workflows individually.
#### Downloading files to the workflow’s `/tmp` directory
First open a new file at a specific path in the File Store, and then call the `toFile()` method to download the file to the given path:
```javascript theme={null}
import fs from 'fs';
export default defineComponent({
async run({ steps, $ }) {
// Download a file from the File Store to the local /tmp/ directory
const file = await $.files.open('recording.mp3').toFile('/tmp/README.md')
// read the file version of the file stored in /tmp
return (await fs.promises.readFile('/tmp/README.md')).toString()
},
})
```
Only the `/tmp/` directory is readable and writable
Make sure that your path to `toFile(path)` includes
### Passing files between steps
Files can be passed between steps. Pipedream will automatically serialize the file as a JSON *description* of the file. Then when you access the file as a step export as a prop in a Node.js code step, then you can interact with the `File` instance directly.
For example, if you have a file stored at the path `logo.png` within your File Store, then within a Node.js code step you can open it:
```javascript theme={null}
// "open_file" Node.js code step
export default defineComponent({
async run({ steps, $ }) {
// Return data to use it in future steps
const file = $.files.open('logo.png')
return file
},
})
```
Then in a downstream code step, you can use it via the `steps` path:
```javascript theme={null}
// "get_file_url" Node.js code step
export default defineComponent({
async run({ steps, $ }) {
// steps.open_file.$return_value is automatically parsed back into a File instance:
return await steps.open_file.$return_value.toUrl()
},
})
```
Files descriptions are compatible with other workflow helpers
Files can also be used with `$.flow.suspend()` and `$.flow.delay()`.
#### Handling lists of files
One limitation of the automatic parsing of files between steps is that it currently doesn’t automatically handle lists of files between steps.
For example, if you have a step that returns an array of `File` instances:
```javascript theme={null}
// "open_files" Node.js code step
export default defineComponent({
async run({ steps, $ }) {
// Return data to use it in future steps
const file1 = $.files.open('vue-logo.svg')
const file2 = $.files.open('react-logo.svg')
return [file1, file]
},
})
```
Then you’ll need to use `$.files.openDescriptor` to parse the JSON definition of the files back into `File` instances:
```typescript theme={null}
// "parse_files" Node.js code step
export default defineComponent({
async run({ steps, $ }) {
const files = steps.open_files.$return_value.map(object => $.files.openDescriptor(object))
// log the URL to the first File
console.log(await files[0].toUrl());
},
})
```
### Deleting files
You can call `delete()` on the file to delete it from the File Store.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Open the file and delete it
const file = await $.files.open('example.png').delete()
console.log('File deleted.')
},
})
```
Deleting files is irreversible
It’s not possible to restore deleted files. Please take care when deleting files.
## FAQ
### Are there size limits for files within File Stores?
At this time no, but File Stores are in preview and are subject to change.
### Are helpers available for Python to download, upload and manage files?
At this time no, only Node.js includes a helper to interact with the File Store programmatically within workflows.
### Are File Stores generally available?
At this time File Stores are only available to Advanced plan and above subscribed workspaces. You can change your plan within the [pricing page](https://pipedream.com/pricing).
# File Stores Node.js Reference
Source: https://pipedream.com/docs/workflows/data-management/file-stores/reference
The File Stores Node.js helper allows you to manage files within Code Steps and Action components.
## `$.files`
The `$.files` helper is the main module to interact with the Project’s File Store. It can instantiate new files, open files from descriptors and list the contents of the File Store.
### `$.files.open(path)`
*Sync.* Opens a file from the relative `path`. If the file doesn’t exist, a new empty file is created.
### `$.files.openDescriptor(fileDescriptor)`
*Sync.* Creates a new `File` from the JSON friendly description of a file. Useful for recreating a `File` from a step export.
For example, export a `File` as a step export which will render the `File` as JSON:
```javascript theme={null}
// create_file
// Creates a new Project File and uploads an image to it
export default defineComponent({
async run({ steps, $ }) {
// create the new file and upload the contents to it from a URL
const file = await $.files.open("imgur.png").fromUrl("https://i.imgur.com/TVIPgNq.png")
// return the file as a step export
return file
},
}
```
Then in a downstream step recreate the `File` instance from the step export friendly *description*:
```javascript theme={null}
// download_file
// Opens a file downloaded from a previous step, and saves it.
export default defineComponent({
async run({ steps, $ }) {
// Convert the the description of the file back into a File instance
const file = $.files.openDescriptor(steps.create_file.$return_value)
// Download the file to the local /tmp directory
await $.file.download('/tmp/example.png')
console.log("File downloaded to /tmp")
},
})
```
### `$.files.dir(?path)`
*Sync.* Lists the files & directories at the given `path`. By default it will list the files at the root directory.
Here’s an example of how to iterate over the files in the root directory and open them as `File` instances:
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// list all contents of the root File Stores directory in this project
const dirs = $.files.dir();
let files = [];
for await(const dir of dirs) {
// if this is a file, let's open it
if(dir.isFile()) {
files.push(await $.files.open(dir.path))
}
}
return files
},
})
```
Each item returned by `$.files.dir()` will contain the following properties:
* `isDirectory()` - `true` if this instance is a directory.
* `isFile()` - `true` if this instance is a file.
* `path` - The path to the file.
* `size` - The size of the file in bytes.
* `modifiedAt` - The last modified at timestamp.
## `File`
This class describes an instance of a single file within a File Store.
When using `$.files.open` or `$.files.openDescriptor`, you’ll create a new instance of a `File` with helper methods that give you more flexibility to perform programmatic actions with the file.
### `File.toUrl()`
*Async.* The pre-signed GET URL to retrieve the file.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Retrieve the pre-signed GET URL for logo.png
const url = await $.files.open('logo.png').toUrl()
return url
},
})
```
Pre-signed GET URLs are short lived.
The `File.toUrl()` will expire after 30 minutes.
### `File.toFile(path)`
*Async.* Downloads the file to the local path in the current workflow. If the file doesn’t exist, a new one will be created at the path specified.
Only `/tmp` is writable in workflow environments
Only the `/tmp` directory is writable in your workflow’s execution environment. So you must download your file to the `/tmp` directory.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Download the file in the File Store to the workflow's /tmp/ directory
await $.files.open('logo.png').toFile("/tmp/logo.png")
},
})
```
### `File.toBuffer()`
*Async.* Downloads the file as a Buffer to create readable or writeable streams.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// opens a file at the path "hello.txt" and downloads it as a Buffer
const buffer = await $.files.open('hello.txt').toBuffer()
// Logs the contents of the Buffer as a string
console.log(buffer.toString())
},
})
```
### `File.fromFile(localFilePath, ?contentType)`
*Async.* Uploads a file from the file at the `/tmp` local path. For example, if `localFilePath` is given `/tmp/recording.mp3`, it will upload that file to the current File Store File instance.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Upload a file to the File Store from the local /tmp/ directory
const file = await $.files.open('recording.mp3').fromFile('/tmp/recording.mp3')
console.log(file.url)
},
})
```
### `File.fromUrl(url)`
*Async.* Accepts a `url` to read from.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Upload a file to the File Store by a URL
const file = await $.files.open('pipedream.png').fromUrl('https://res.cloudinary.com/pipedreamin/image/upload/t_logo48x48/v1597038956/HzP2Yhq8_400x400_1_sqhs70.jpg')
console.log(file.url)
},
})
```
### `File.createWriteStream(?contentType, ?contentLength)`
*Async.* Creates a write stream to populate the file with.
Pass the content length if possible
The `contentLength` argument is optional, however we do recommend passing it. Otherwise the entire file will need to be written to the local `/tmp` before it can be uploaded to the File store.
```javascript theme={null}
import { pipeline } from 'stream/promises';
import got from 'got'
export default defineComponent({
async run({ steps, $ }) {
const writeStream = await $.files.open('logo.png').createWriteStream("image/png", 2153)
const readStream = got.stream('https://pdrm.co/logo')
await pipeline(readStream, writeStream);
},
})
```
### `File.fromReadableStream(?contentType, ?contentLength)`
*Async.* Populates a file’s contents from the `ReadableStream`.
Pass the content length if possible
The `contentLength` argument is optional, however we do recommend passing it. Otherwise the entire file will need to be written to the local `/tmp` before it can be uploaded to the File store.
```javascript theme={null}
import got from 'got'
export default defineComponent({
async run({ steps, $ }) {
// Start a new read stream
const readStream = got.stream('https://pdrm.co/logo')
// Populate the file's content from the read stream
await $.files.open("logo.png").fromReadableStream(readStream, "image/png", 2153)
},
})
```
### `File.delete()`
*Async.* Deletes the Project File.
```javascript theme={null}
export default defineComponent({
async run({ steps, $ }) {
// Open the Project File and delete it
const file = await $.files.open('example.png').delete()
console.log('File deleted.')
},
})
```
Deleting files is irreversible
It’s not possible to restore deleted files. Please take care when deleting files.
# Custom Domains
Source: https://pipedream.com/docs/workflows/domains
By default, all new [Pipedream HTTP endpoints](/docs/workflows/building-workflows/triggers/#http) are hosted on the domain. But you can configure any domain you want: instead of `https://endpoint.m.pipedream.net`, the endpoint would be available on `https://endpoint.example.com`.
## Configuring a new custom domain
### 1. Choose your domain
You can configure any domain you own to work with Pipedream HTTP endpoints. For example, you can host Pipedream HTTP endpoints on a dedicated subdomain on your core domain, like `*.eng.example.com` or `*.marketing.example.com`. This can be any domain or subdomain you own.
In this example, endpoints would look like:
```txt theme={null}
[endpoint_id].eng.example.com
[endpoint_id_1].eng.example.com
...
```
where `[endpoint_id]` is a uniquely-generated hostname specific to your Pipedream HTTP endpoint.
If you own a domain that you want to *completely* redirect to Pipedream, you can also configure `*.example.com` to point to Pipedream. In this example, endpoints would look like:
```txt theme={null}
[endpoint_id].example.com
[endpoint_id_1].example.com
...
```
Since all traffic on `*.example.com` points to Pipedream, we can assign hosts on the root domain. This also means that **you cannot use other hosts like [www.example.com](http://www.example.com)** without conflicting with Pipedream endpoints. Choose this option only if you’re serving all traffic from `example.com` from Pipedream.
Before you move on, make sure you have access to manage DNS records for your domain. If you don’t, please coordinate with the team at your company that manages DNS records, and feel free to [reach out to our Support team](https://pipedream.com/support) with any questions.
#### A note on domain wildcards
Note that the records referenced above use the wildcard (`*`) for the host portion of the domain. When you configure DNS records in [step 3](/docs/workflows/domains/#3-add-your-dns-records), this allows you to point all traffic for a specific domain to Pipedream and create any number of Pipedream HTTP endpoints that will work with your domain.
### 2. Reach out to Pipedream Support
Once you’ve chosen your domain and are in an [eligible plan](https://pipedream.com/pricing), [reach out to Pipedream Support](https://pipedream.com/support) and let us know what domain you’d like to configure for your workspace. We’ll configure a TLS/SSL certificate for that domain, and give you two DNS CNAME records to add for that domain in [step 3](/docs/workflows/domains/#3-add-your-dns-records).
### 3. Add your DNS records
Once we configure your domain, we’ll ask you to create two DNS CNAME records:
* [One record to prove ownership of your domain](/docs/workflows/domains/#add-the-cname-validation-record) (a `CNAME` record)
* [Another record to point traffic on your domain to Pipedream](/docs/workflows/domains/#add-the-dns-cname-wildcard-record) (a `CNAME` record)
#### Add the CNAME validation record
Pipedream uses [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/) to create the TLS certificate for your domain. To validate the certificate, you need to add a specific DNS record provided by Certificate Manager. Pipedream will provide the name and value.
For example, if you requested `*.eng.example.com` as your custom domain, Pipedream will provide the details of the record, like in this example:
* **Type**: `CNAME`
* **Name**: `_2kf9s72kjfskjflsdf989234nsd0b.eng.example.com`
* **Value**: `_7ghslkjsdfnc82374kshflasfhlf.vvykbvdtpk.acm-validations.aws.`
* **TTL (seconds)**: 300
Consult the docs for your DNS service for more information on adding CNAME records. Here’s an example configuration using AWS’s Route53 service:
![]()
#### Add the DNS CNAME wildcard record
Now you’ll need to add the wildcard record that points all traffic for your domain to Pipedream. Pipedream will also provide the details of this record, like in this example:
* **Type**: `CNAME`
* **Name**: `*.eng.example.com`
* **Value**: `id123.cd.pdrm.net.`
* **TTL (seconds)**: 300
Once you’ve finished adding these DNS records, please **reach out to the Pipedream team**. We’ll validate the records and finalize the configuration for your domain.
### 4. Send a test request to your custom domain
Any traffic to existing endpoints will continue to work uninterrupted.
To confirm traffic to your new domain works, take any Pipedream endpoint URL and replace the with your custom domain. For example, if you configured a custom domain of `pipedream.example.com` and have an existing endpoint at
```txt theme={null}
https://[endpoint_id].m.pipedream.net
```
Try making a test request to
```txt theme={null}
https://[endpoint_id].eng.example.com
```
## Security
### How Pipedream manages the TLS/SSL certificate
See our [TLS/SSL security docs](/docs/privacy-and-security/#encryption-of-data-in-transit-tls-ssl-certificates) for more detail on how we create and manage the certificates for custom domains.
### Requests to custom domains are allowed only for your endpoints
Custom domains are mapped directly to customer endpoints. This means no other customer can send requests to their endpoints on *your* custom domain. Requests to `example.com` are restricted specifically to HTTP endpoints in your Pipedream workspace.
# Environment Variables
Source: https://pipedream.com/docs/workflows/environment-variables
Environment variables (env vars) enable you to separate secrets and other static configuration data from your code.
You shouldn’t include API keys or other sensitive data directly in your workflow’s code. By referencing the value of an environment variable instead, your workflow includes a reference to that variable — for example, `process.env.API_KEY` instead of the API key itself.
You can reference env vars and secrets in [workflow code](/docs/workflows/building-workflows/code/) or in the object explorer when passing data to steps, and you can define them either globally for the entire workspace, or scope them to individual projects.
| Scope | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Workspace** | All environment variables are available to all workflows within the workspace. All workspace members can manage workspace-wide variables [in the UI](https://pipedream.com/settings/env-vars). |
| **Project** | Environment variables defined within a project are only accessible to the workflows within that project. Only workspace members who have [access to the project](/docs/projects/access-controls/) can manage project variables. |
## Creating and updating environment variables
* To manage **global** environment variables for the workspace, navigate to **Settings**, then click **Environment Variables**: [https://pipedream.com/settings/env-vars](https://pipedream.com/settings/env-vars)
* To manage environment variables within a project, open the project, then click **Variables** from the project nav on the left
![]()
Click **New Variable** to add a new environment variable or secret:
![]()
**Configure the required fields**:
![]()
| Input field | Description |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **Key** | Name of the variable — for example, `CLIENT_ID` |
| **Value** | The value, which can contain any string with a max limit of 64KB |
| **Description** | Optionally add a description of the variable. This is only visible in the UI, and is not accessible within a workflow. |
| **Secret** | New variables default to **secret**. If configured as a secret, the value is never exposed in the UI and cannot be modified. |
To edit an environment variable, click the **Edit** button from the three dots to the right of a specific variable.
![]()
* Updates to environment variables will be made available to your workflows as soon as the save operation is complete — typically a few seconds after you click **Save**.
* If you update the value of an environment variable in the UI, your workflow should automatically use that new value where it’s referenced.
* If you delete a variable in the UI, any deployed workflows that reference it will return `undefined`.
## Referencing environment variables in code
You can reference the value of any environment variable using the object [`process.env`](https://nodejs.org/dist/latest-v10.x/api/process.html#process_process_env). This object contains environment variables as key-value pairs.
For example, let’s say you have an environment variable named `API_KEY`. You can reference its value in Node.js using `process.env.API_KEY`:
```javascript theme={null}
const url = `http://yourapi.com/endpoint/?api_key=${process.env.API_KEY}`;
```
Reference the same environment variable in Python:
```python theme={null}
import os
print(os.environ["API_KEY"])
```
Variable names are case-sensitive. Use the key name you defined when referencing your variable in `process.env`.
Referencing an environment variable that doesn’t exist returns the value `undefined` in Node.js. For example, if you try to reference `process.env.API_KEY` without first defining the `API_KEY` environment variable in the UI, it will return the value `undefined`.
### Using autocomplete to reference env vars
When referencing env vars directly in code within your Pipedream workflow, you can also take advantage of autocomplete:
![]()
Logging the value of any environment variables — for example, using `console.log` — will include that value in the logs associated with the cell. Please keep this in mind and take care not to print the values of sensitive secrets.
`process.env` will always return `undefined` when used outside of the `defineComponent` export.
## Referencing environment variables in actions
[Actions](/docs/components/contributing/#actions) are pre-built code steps that let you provide input in a form, selecting the correct params to send to the action.
You can reference the value of environment variables using `{{process.env.YOUR_ENV_VAR}}`. You’ll see a list of your environment variables in the object explorer when selecting a variable to pass to a step.
![]()
[Private components](/docs/components/contributing/#using-components) (actions or triggers) do not have direct access to workspace or project variables as public components or code steps. Add a prop specifically for the variable you need. For sensitive data like API keys, [configure the prop as a secret](/docs/components/contributing/api/#props). In your prop configuration, set the value to `{{process.env.YOUR_ENV_VAR}}` to securely reference the environment variable.
## FAQ
### What if I define the same variable key in my workspace env vars and project env vars?
The project-scoped variable will take priority if the same variable key exists at both the workspace and project level. If a workflow *outside* of the relevant project references that variable, it’ll use the value of the environment variable defined for the workspace.
### What happens if I share a workflow that references an environment variable?
If you [share a workflow](/docs/workflows/building-workflows/sharing/) that references an environment variable, **only the reference is included, and not the actual value**.
## Limits
* Currently, environment variables are only exposed in Pipedream workflows, [not event sources](https://github.com/PipedreamHQ/pipedream/issues/583).
* The value of any environment variable may be no longer than `64KB`.
* The names of environment variables must start with a letter or underscore.
* Pipedream reserves environment variables that start with `PIPEDREAM_` for internal use. You cannot create an environment variable that begins with that prefix.
# Event History
Source: https://pipedream.com/docs/workflows/event-history
Monitor all workflow events and their stack traces in one centralized view under the [**Event History**](https://pipedream.com/event-history) section in the dashboard.
Within the **Event History**, you’ll be able to filter your events by workflow, execution status, within a specific time range.
Workspace admins are able to view events for all workflows, but members are required to select a workflow since they might not have [access to certain projects](/docs/projects/access-controls/#permissions).
## Filtering Events
The filters at the top of the screen allow you to search all events processed by your workflows.
You can filter by the event’s **Status**, **time of initiation** or by the **Workflow name**.
The filters are scoped to the current [workspace](/docs/workspaces/). If you’re not seeing the events or workflow you’re expecting, try [switching workspaces](/docs/workspaces/#switching-between-workspaces).
### Filtering by status
* The **Status** filter controls which events are shown by their status
* For example selecting the **Success** status, you’ll see all workflow events that were successfully executed
![]()
#### All failed workflow executions
* You can view all failed workflow executions by applying the **Error** status filter
* This will only display the failed workflow executions in the selected time period
* This view in particular is helpful for identifying trends of errors, or workflows with persistent problems
![]()
#### All paused workflow executions
* Workflow executions that are currently in a suspended state from `$.flow.delay` or `$.flow.suspend` will be shown when this filter is selected
![]()
If you’re using `setTimeout` or `sleep` in Node.js or Python steps, the event will not be considered **Paused**. Using those language native execution holding controls leaves your workflow in a **Executing** state.
### Within a timeframe
* Filtering by time frame will include workflow events that *began* execution within the defined range
* Using this dropdown, you can select between convenient time ranges, or specify a custom range on the right side
![]()
### Filtering by workflow
You can also filter events by a specific workflow. You can search by the workflow’s name in the search bar in the top right.
![]()
Alternatively, you can filter by workflow from a specific event. First, open the menu on the far right, then select **Filter By Workflow**. Then only events processed by that workflow will appear.
![]()
## Inspecting events
* Clicking on an individual event will open a panel that displays the steps executed, their individual configurations, as well as the overall performance and results of the entire workflow.
* The top of the event history details will display details including the overall status of that particular event execution and errors if any.
* If there is an error message, the link at the bottom of the error message will link to the corresponding workflow step that threw the error.
* From here you can easily **Build from event** or **Replay event**
![]()
## Bulk actions
You can select multiple events and perform bulk actions on them.
* **Replay**: Replays the selected events. This is useful for example when you have multiple errored events that you want to execute again after fixing a bug in your workflow.
* **Delete**: Deletes the selected events. This may be useful if you have certain events you want to scrub from the event history, or when you’ve successfully replayed events that had originally errored.
When you replay multiple events at once, they’ll be replayed in the order they were originally executed. This means the first event that came in will be replayed first, followed by the second, and so on.
## Limits
The number of events recorded and available for viewing in the Event History depends on your plan. [Please see the pricing page](https://pipedream.com/pricing) for more details.
## FAQ
### Is Event History available on all plans?
Yes, event history is available for all workspace plans, including free plans. However, the length of searchable or viewable history changes depending on your plan. [Please see the pricing page](https://pipedream.com/pricing) for more details.
# GitHub Sync
Source: https://pipedream.com/docs/workflows/git
When GitHub Syncing is enabled on your project, Pipedream will serialize your workflows and synchronize changes to a GitHub repo.
Capabilities include:
* Bi-directional GitHub sync (push and pull changes)
* Edit in development branches
* Track commit and merge history
* Link users to commits
* Merge from Pipedream or create PRs and merge from GitHub
* Edit in Pipedream or use a local editor and synchronize via GitHub (e.g., edit code, find and replace across multiple steps or workflows)
* Organize workflows into projects with support for nested folders
## Getting Started
### Create a new project and enable GitHub Sync
A project may contain one or more workflows and may be further organized using nested folders. Each project may be synchronized to a single GitHub repo.
* Go to [https://pipedream.com/projects](https://pipedream.com/projects)
* Create a new project
* Enter a project name and check the box to **Configure GitHub Sync**
* To use **OAuth**
* Select a connected account, GitHub scope and repo name
* Pipedream will automatically create a new, empty repo in GitHub
![]()
* To use **Deploy Keys**
* [Create a new repo](https://github.com/new) in GitHub
* Copy and paste the URL of your new repo
* Follow the instructions to configure the deploy key
* Test your setup and create a new project
![]()
### Create a branch to edit a project
Branches are required to make changes
All changes to resources in a project must be made in a development branch.
Examples of changes include creating, editing, deleting, enabling, disabling and renaming workflows. This also includes changing workflow settings like concurrency, VPC assignment and auto-retries.
To edit a git-backed project you must create a development branch by clicking **Edit > Create Branch**
![]()
Next, name the branch and click **Create**:
![]()
To exit development mode without merging to production, click **Exit Development Mode**:
![]()
Your changes will be saved to the branch, if you choose to revisit them later.
### Merge changes to production
Once you’ve committed your changes, you can deploy your changes by merging them into the `production` branch through the Pipedream UI or GitHub.
When you merge a Git-backed project to production, all modified resources in the project will be deployed. Multiple workflows may be deployed, modified, or deleted in production through a single merge action.
#### Merge via the Pipedream UI
To merge changes to production, click on **Merge to production:**
![]()
Pipedream will present a diff between the development branch and the `production` branch. Validate your changes and click **Merge to production** to complete the merge:
![]()
#### Create a Pull Request in GitHub
To create a pull request in GitHub, either choose **Open GitHub pull request** from the git-actions menu in Pipedream or in GitHub:
![]()
You can also review and merge changes directly from GitHub using the standard pull request process.
Pull request reviews cannot be required. That feature is on the roadmap for the Business tier.
### Commit changes
To commit changes without merging to production, select **Commit Changes** from the Git Actions menu:
![]()
You can review the diff and enter a commit message:
![]()
### Pull changes and resolve conflicts
If remote changes are detected, you’ll be prompted to pull the changes:
![]()
Pipedream will attempt to automatically merge changes. If there are conflicts, you will be prompted to manually resolve it:
![]()
### Move existing workflows to projects
Legacy (v1) workflows are not supported in projects.
First, select the workflow(s) you want to move from the [workflows listing page](https://pipedream.com/workflows) and click **Move** in the top action menu:
![]()
Then, select the project to move the selected workflows to:
![]()
Undeployed changes are automatically assigned a development branch
If any moved workflows have undeployed changes, those changes will staged in a branch prefixed with `undeployed-changes` (e.g., `undeployed-changes-27361`).
### Use the changelog
The changelog tracks all git activity (for projects with GitHub sync enabled). If you encounter an error merging your project, go to the changelog and explore the log details to help you troubleshoot issues in your workflows:
![]()
### Local development
Projects that use GitHub sync may be edited outside of Pipedream. You can edit and commit directly via GitHub’s UI or clone the repo locally and use your preferred editor (e.g., VSCode).
To test external edits in Pipedream:
1. Commit and push local changes to your development branch in GitHub
2. Open the project in Pipedream’s UI and load your development branch
3. Use the Git Actions menu to pull changes from GitHub
## Known Issues
Below are a list of known issues that do not currently have solutions, but are in progress:
* Project branches on Pipedream cannot be deleted.
* If a workflow uses an action that has been deprecated, merging to production will fail.
* Legacy (v1) workflows are not supported in projects.
* Self-hosted GitHub Server instances are not yet supported. [Please contact us for help](https://pipedream.com/support).
* Workflow attachments are not supported
## GitHub Enterprise Cloud
If your repository is hosted on a GitHub Enterprise account, you can allow Pipedream’s address range to sync your project changes.
[Follow the directions here](https://docs.github.com/en/enterprise-cloud@latest/organizations/keeping-your-organization-secure/managing-security-settings-for-your-organization/managing-allowed-ip-addresses-for-your-organization) and add the following IP range:
To use this public IP address and connect to GitHub Enterprise Cloud hosted repositories, you’ll need to have a Pipedream Business plan. [View our plans](https://pipedream.com/pricing).
## FAQ
### How are Pipedream workflows synchronized to GitHub?
Pipedream will serialize your project’s workflows and their configuration into a standard YAML format for storage in GitHub.
Then Pipedream will commit your changes to your connected GitHub repository.
### Do you have a definition of this YAML?
Not yet, please stay tuned!
### Can I sync multiple workflows to a single GitHub repository?
Yes, *projects* are synced to a single GitHub repository which allows you to store multiple workflows into a single GitHub repository for easier organization and management.
### Can I use this feature to develop workflows locally?
Yes, you can use the GitHub Syncing feature to develop your workflows from YAML files checked into your Pipedream connected GitHub repository.
Then pushing changes to the `production` branch will trigger a deploy for your Pipedream workflows.
### Why am I seeing the error “could not resolve step\[index].uses: component-key\@version” when merging to production?
This error occurs when a workflow references a [private component](/docs/components/contributing/#using-private-actions) without properly prefixing the component key with your workspace name in the `workflow.yaml` configuration file. Pipedream requires this prefix to correctly identify and resolve components specific to your workspace.
For example, if you modified a [registry action](/docs/components/contributing/) and published it privately, the correct component key should be formatted as `@workspacename/component-key@version` (e.g., `@pipedream/github-update-issue@0.1.0`).
To resolve this error:
1. Clone your repository locally and create a development branch.
2. Locate the error in your `workflow.yaml` file where the component key is specified.
3. Add your workspace name prefix to the component key, ensuring it follows the format `@workspacename/component-key@version`.
4. Commit your changes and push them to your repository.
5. Open your project in the Pipedream UI and select your development branch.
6. Click on **Merge to Production** and verify the deployment success in the [Changelog](/docs/workflows/git/#use-the-changelog).
7. If the issue persists, [reach out to Pipedream Support](https://pipedream.com/support) for further assistance.
### Why am I seeing an error about “private auth mismatch” when trying to merge a branch to production?
![]()
This error occurs when **both** of the below conditions are met:
1. The referenced workflow is using a connected account that’s not shared with the entire workspace
2. The change was merged from outside the Pipedream UI (via github.com or locally)
Since Pipedream can’t verify the person who merged that change should have access to use the connected account in a workflow in this case, we block these deploys.
To resolve this error:
1. Make sure all the connected accounts in the project’s workflows are [accessible to the entire workspace](/docs/apps/connected-accounts/#access-control)
2. Re-trigger a sync with Pipedream by making a nominal change to the workflow **from outside the Pipedream UI** (via github.com or locally), then merge that change to production
### Can I sync an existing GitHub repository with workflows to a new Pipedream Project?
No, at this time it’s not possible because of how resources are connected during the bootstrapping process from the workflow YAML specification. However, this is on our roadmap, [please subscribe to this issue](https://github.com/PipedreamHQ/pipedream/issues/9255) for the latest details.
### Migrating GitHub Repositories
You can migrate Pipedream project’s GitHub repository to a new repository, while preserving history. You may want to do this when migrating a repository from a personal GitHub account to an organization account, without affecting the workflows within the Pipedream project.
#### Assumptions
* **Current GitHub repository**: `previous_github_repo`
* **New GitHub repository**: `new_github_repo`
* Basic familiarity with git and GitHub
* Access to a local terminal (e.g., Bash, Zsh, PowerShell)
* Necessary permissions to modify both the Pipedream project and associated GitHub repositories
#### Steps
1. **Access Project Settings in Pipedream:**
* Navigate to your Pipedream project.
* Use the dropdown menu on the “Edit” button in the top right corner to access `previous_github_repo` in GitHub.
![]()
2. **Clone the Current Repository Locally:**
```bash theme={null}
git clone previous_github_repo_clone_url
```
3. Reset GitHub Sync in Pipedream:
* In Pipedream, go to your project settings.
* Click on “Reset GitHub Connection”.
![]()
4. Set Up New repository connection:
* Configure the project’s GitHub repository to use `new_github_repo`.
5. Clone the new repository locally:
```bash theme={null}
git clone new_github_repo_clone_url
cd new_github_repo
```
6. Link to the old repository:
```bash theme={null}
git remote add old_github_repo previous_github_repo_clone_url
git fetch --all
```
7. Prepare for migration:
* Create and switch to a new branch for migration:
```bash theme={null}
git checkout -b migration
```
* Merge the main branch of `old_github_repo` into migration, allowing for unrelated histories:
```bash theme={null}
git merge --allow-unrelated-histories old_github_repo/production
# Resolve any conflicts, such as in README.md
git commit
```
8. Finalize the migration:
* Optionally push the `migration` branch to the remote:
```bash theme={null}
git push --set-upstream origin migration
```
* Switch to the `production` branch and merge:
```bash theme={null}
git checkout production
git merge --no-ff migration
git push
```
9. Cleanup:
* Remove the connection to the old repository:
```bash theme={null}
git remote remove old_github_repo
```
* Optionally, you may now safely delete `previous_github_repo` from GitHub.
### How does the `production` branch work?
Anything merged to the `production` branch will be deployed to your production workflows on Pipedream.
From a design perspective, we want to let you manage any branching strategy on your end, since you may be making commits to the repo outside of Pipedream. Once we support managing Pipedream workflows in a monorepo, where you may have other changes, we wanted to use a branch that didn’t conflict with a conventional main branch (like `main` or `master`).
In the future, we also plan to support you changing the default branch name.
# Limits
Source: https://pipedream.com/docs/workflows/limits
Pipedream imposes limits on source and workflow execution, the events you send to Pipedream, and other properties. You’ll receive an error if you encounter these limits. See our [troubleshooting guide](/docs/troubleshooting/) for more information on these specific errors.
Some of these limits apply only on the free tier. For example, Pipedream limits the number of credits and active workflows you can use on the free tier. **On paid tiers, you can run an unlimited number of credits for any amount of execution time (usage charges apply)**.
Other limits apply across the free and paid tiers. Please see the details on each limit below.
**These limits are subject to change at any time**.
## Number of Workflows
The limit of active workflows depends on your current plan. [See our pricing page](https://pipedream.com/pricing) for more details.
## Number of Event Sources
**You can run an unlimited number of event sources**, as long as each operates under the limits below.
## Execution Credits
Free Pipedream account have a limit on the number of execution credits. Paid plans are not capped but are subject to additional usage charges (you can manually set a usage cap if you want to manage costs).\
\
You can view your credits usage at the bottom-left of [the Pipedream UI](https://pipedream.com).
You can also see more detailed usage in [Billing and Usage Settings](https://pipedream.com/settings/billing). Here you’ll find your usage for the last 30 days, broken out by day, by resource (e.g. your source / workflow).
### Included Credits Usage Notifications
| Tier | Notifications |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Free tiers | You’ll receive an email when you reach 100% of your usage. |
| Paid tiers | You’ll receive an email at 80% and 100% of your [included credits](/docs/pricing/#included-credits) for your [billing period](/docs/pricing/#billing-period). |
## Daily workflow testing limit
You **do not** use credits testing workflows, but workspaces on the **Free** plan are limited to of test runtime per day. If you exceed this limit when testing in the builder, you’ll see a **Runtime Quota Exceeded** error.
## Data stores
Depending on your plan, Pipedream sets limits on:
1. The total number of data stores
2. The total number of keys across all data stores
3. The total storage used across all data stores
You’ll find your workspace’s limits in the **Data Stores** section of usage dashboard in the bottom-left of [the Pipedream UI](https://pipedream.com).
![]()
## HTTP Triggers
The following limits apply to [HTTP triggers](/docs/workflows/building-workflows/triggers/#http).
### HTTP Request Body Size
By default, the body of HTTP requests sent to a source or workflow is limited to .
Your endpoint will issue a `413 Payload Too Large` status code when the body of your request exceeds .
**Pipedream supports two different ways to bypass this limit**. Both of these interfaces support uploading data up to `5TB`, though you may encounter other platform limits.
* You can send large HTTP payloads by passing the `pipedream_upload_body=1` query string or an `x-pd-upload-body: 1` HTTP header in your HTTP request. [Read more here](/docs/workflows/building-workflows/triggers/#sending-large-payloads).
* You can upload multiple large files, like images and videos, using the [large file upload interface](/docs/workflows/building-workflows/triggers/#large-file-support).
### QPS (Queries Per Second)
Generally the rate of HTTP requests sent to an endpoint is quantified by QPS, or *queries per second*. A query refers to an HTTP request.
**You can send an average of 10 requests per second to your HTTP trigger**. Any requests that exceed that threshold may trigger rate limiting. If you’re rate limited, we’ll return a `429 Too Many Requests` response. If you control the application sending requests, you should retry the request with [exponential backoff](https://cloud.google.com/storage/exponential-backoff) or a similar technique.
We’ll also accept short bursts of traffic, as long as you remain close to an average of 10 QPS (e.g. sending a batch of 50 requests every 30 seconds should not trigger rate limiting).
**This limit can be raised for paying customers**. To request an increase, [reach out to our Support team](https://pipedream.com/support/) with the HTTP endpoint whose QPS you’d like to increase, with the new, desired limit.
## Email Triggers
Currently, most of the [limits that apply to HTTP triggers](/docs/workflows/limits/#http-triggers) also apply to [email triggers](/docs/workflows/building-workflows/triggers/#email).
The only limit that differs between email and HTTP triggers is the payload size: the total size of an email sent to a workflow - its body, headers, and attachments - is limited to .
## Memory
By default, workflows run with of memory. You can modify a workflow’s memory [in your workflow’s Settings](/docs/workflows/building-workflows/settings/#memory), up to .
Increasing your workflow’s memory gives you a proportional increase in CPU. If your workflow is limited by memory or compute, increasing your workflow’s memory can reduce its overall runtime and make it more performant.
**Pipedream charges credits proportional to your memory configuration**. [Read more here](/docs/pricing/faq/#how-does-workflow-memory-affect-credits).
## Disk
Your code, or a third party library, may need access to disk during the execution of your workflow or event source. **You have access to of disk in the `/tmp` directory**.
This limit cannot be raised.
## Workflows
### Time per execution
Every event sent to a workflow triggers a new execution of that workflow. Workflows have a default execution limit that varies with the trigger type:
* HTTP and Email-triggered workflows default to **30 seconds** per execution.
* Cron-triggered workflows default to **60 seconds** per execution.
If your code exceeds your workflow-level limit, we’ll throw a **Timeout** error and stop your workflow. Any partial logs and observability associated with code cells that ran successfully before the timeout will be attached to the event in the UI, so you can examine the state of your workflow and troubleshoot where it may have failed.
You can increase the timeout limit, up to a max value set by your plan:
| Tier | Maximum time per execution |
| ---------- | -------------------------- |
| Free tiers | 300 seconds (5 min) |
| Paid tiers | 750 seconds (12.5 min) |
Events that trigger a **Timeout** error will appear in red in the [Inspector](/docs/workflows/building-workflows/inspect/). You’ll see the timeout error, also in red, in the cell at which the code timed out.
### Event History
The [Inspector](/docs/workflows/building-workflows/inspect/#the-inspector) shows the execution history for a given workflow. Events have a limited retention period, depending on your plan:
| Tier | Events retained per workflow |
| ---------- | -------------------------------------------------------------------------------- |
| Free tiers | |
| Paid tiers | [View breakdown of events history per paid plan](https://pipedream.com/pricing/) |
The execution details for a specific event also expires after days.
### Logs, Step Exports, and other observability
The total size of `console.log()` statements, [step exports](/docs/workflows/#step-exports), and the original event data sent to the workflow cannot exceed a combined size of . If you produce logs or step exports larger than this - for example, passing around large API responses, CSVs, or other data - you may encounter a **Function Payload Limit Exceeded** in your workflow.
This limit cannot be raised.
## Acceptable Use
We ask that you abide by our [Acceptable Use](https://pipedream.com/terms/#b-acceptable-use) policy. In short this means: don’t use Pipedream to break the law; don’t abuse the platform; and don’t use the platform to harm others.
# Workflow Development
Source: https://pipedream.com/docs/workflows/quickstart
Sign up for a [free Pipedream account](https://pipedream.com/auth/signup) (no credit card required) and complete this quickstart guide to learn the basic patterns for workflow development:
Workflows must be created in **Projects**. Projects make it easy to organize your workflows and collaborate with your team.
Go to [https://pipedream.com/projects](https://pipedream.com/projects) and click on **Create Project**.
![]()
Next, enter a project name and click **Create Project**. For this example, we’ll name our project **Getting Started**. You may also click the icon to the right to generate a random project name.
![]()
[Configure GitHub Sync](/docs/workflows/git/) for projects to enable git-based version control and unlock the ability to develop in branches, commit to or pull changes from GitHub, view diffs, create PRs and more.
After the project is created, use the **New** button to create a new workflow.
![]()
Name the workflow and click **Create Workflow** to use the default settings. For this example, we’ll name the workflow **Pipedream Quickstart**.
![]()
Next, Pipedream will launch the workflow builder and prompt you to add a trigger.
![]()
Clicking the trigger opens a new menu to select the trigger. For this example, select **New HTTP / Webhook Requests**.
![]()
Click **Save and continue** in the step editor on the right to accept the default settings.
![]()
Pipedream will generate a unique URL to trigger this workflow. Once your workflow is deployed, your workflow will run on every request to this URL.
![]()
Next, generate a test event to help you build the workflow.
The test event will be used to provide autocomplete suggestion as you build your workflow. The data will also be used when testing later steps. You may generate or select a different test event at any time when building a workflow.
For this example, let’s use the following test event data:
```json theme={null}
{
"message": "Pipedream is awesome!"
}
```
Pipedream makes it easy to generate test events for your HTTP trigger. Click on **Generate Test Event** to open the HTTP request builder. Copy and paste the JSON data above into the **Raw Request Body** field and click **Send HTTP Request**.
![]()
Pipedream will automatically select and display the contents of the selected event. Validate that the `message` was received as part the event `body`.
![]()
You may also send live data to the unique URL for your workflow using your favorite HTTP tool or by running a `cURL` command, e.g.,
```bash theme={null}
curl -d '{"message": "Pipedream is awesome!"}' \
-H "Content-Type: application/json" \
YOUR_ENDPOINT_URL
```
Before we send data to Google Sheets, let’s use the npm [`sentiment`](https://www.npmjs.com/package/sentiment) package to generate a sentiment score for our message. To do that, click **Continue** or the **+** button.
![]()
That will open the **Add a step** menu. Select **Run custom code**.
![]()
Pipedream will add a Node.js code step to the workflow.
![]()
Rename the step to **sentiment**.
![]()
Next, add the following code to the code step:
```javascript theme={null}
import Sentiment from "sentiment"
export default defineComponent({
async run({ steps, $ }) {
let sentiment = new Sentiment()
return sentiment.analyze(steps.trigger.event.body.message)
},
})
```
This code imports the npm package, passes the message we sent to our trigger to the `analyze()` function by referencing `steps.trigger.event.body.message` and then returns the result.
To use any npm package on Pipedream, just `import` it. There’s no `npm install` or `package.json` required.
Any data you `return` from a step is exported so it can be inspected and referenced it in future steps via the `steps` object. In this example, return values will be exported to `steps.sentiment.$return_value` because we renamed the step to **sentiment** .
Your code step should now look like the screenshot below. To run the step and test the code, click the **Test** button.
![]()
You should see the results of the sentiment analysis when the test is complete.
![]()
When you **Test** a step, only the current step is executed. Use the caret to test different ranges of steps including the entire workflow.
Next, create a Google Sheet and add **Timestamp**, **Message** and **Sentiment Score** to the first row. These labels act as our column headers and will help us configure the Google Sheets step of the workflow.
![]()
Next, let’s add a step to the workflow to send the data to Google Sheets. First, click **+** after the `sentiment` code step and select the **Google Sheets** app.
![]()
Then select the **Add Single Row** action.
![]()
Click to connect you Google Sheets account to Pipedream (or select it from the dropdown if you previously connected an account).
![]()
Pipedream will open Google’s sign in flow in a new window. Sign in with the account you want to connect.
![]()
If prompted, you must check the box for Pipedream to **See, edit, create and delete all of your Google Drive files**. These permissions are required for configure and use the pre-built actions for Google Sheets.
![]()
Learn more about Pipedream’s [privacy and security policy](/docs/privacy-and-security/).
When you complete connecting your Google account, the window should close and you should return to Pipedream. Your connected account should automatically be selected. Next, select your spreadsheet from the dropdown menu:
![]()
Then select the sheet name (the default sheet name in Google Sheets is **Sheet1**):
![]()
Next, select if the spreadsheet has headers in the first row. When a header row exists, Pipedream will automatically retrieve the header labels to make it easy to enter data (if not, you can manually construct an array of values). Since the sheet for this example contains headers, select **Yes**.
![]()
Pipedream will retrieve the headers and generate a form to enter data in your sheet:
![]()
First, let’s use the object explorer to pass the timestamp for the workflow event as the value for the first column. This data can be found in the context object on the trigger.
When you click into the **Timestamp** field, Pipedream will display an object explorer to make it easy to find data. Scroll or search to find the `ts` key under `steps.trigger.context`.
![]()
Click **select path** to insert a reference to `steps.trigger.context.ts`:
![]()
Next, let’s use autocomplete to enter a value for the **Message** column. First, add double braces `{{` — Pipedream will automatically add the closing braces `}}`.
Then, type `steps.trigger.event.body.message` between the pairs of braces. Pipedream will provide autocomplete suggestions as you type. Press **Tab** to use a suggestion and then click `.` to get suggestions for the next key. The final value in the **Message** field should be `steps.trigger.event.body.message`.
![]()
Finally, let’s copy a reference from a previous step. Click on the `sentiment` step to open the results in the editor:
![]()
Next, click the **Copy Path** link next to the score.
![]()
Click the Google Steps step or click the open tab in the editor. Then paste the value into the **Sentiment Score** field — Pipedream will automatically wrap the reference in double braces `{{ }}`.
![]()
Now that the configuration is complete, click **Test** to validate the configuration for this step. When the test is complete, you will see a success message and a summary of the action performed:
![]()
If you load your spreadsheet, you should see the data Pipedream inserted.
![]()
Next, return to your workflow and click **Deploy** to run your workflow on every trigger event.
![]()
When your workflow deploys, you will be redirected to the **Inspector**. Your workflow is now live.
![]()
To validate your workflow is working as expected, send a new request to your workflow: You can edit and run the following `cURL` command:
```bash theme={null}
curl -d '{ "message": "Pipedream is awesome!" }' \
-H "Content-Type: application/json" \
YOUR-TRIGGER-URL
```
The event will instantly appear in the event list. Select it to inspect the workflow execution.
![]()
Finally, you can return to Google Sheets to validate that the new data was automatically inserted.
![]()
## Next Steps
Congratulations! You completed the quickstart and should now understand the basic patterns for workflow development. Next, try creating your own [workflows](/docs/workflows/building-workflows/), learn how to [build and run workflows for your users](/docs/connect/workflows/) or check out the rest of the [docs](/docs/)!
# Virtual Private Clouds
Source: https://pipedream.com/docs/workflows/vpc
Pipedream VPCs enable you to run workflows in dedicated and isolated networks with static outbound egress IP addresses that are unique to your workspace (unlike other platforms that provide static IPs common to all customers on the platform).
Outbound network requests from workflows that run in a VPC will originate from these static IP addresses, so you can whitelist access to sensitive resources (like databases and APIs) with confidence that the requests will only originate from the Pipedream workflows in your workspace.
Looking for static egress IPs for **Connect** tool calls (action executions)? See [Virtual Private Clouds for Connect](/docs/connect/vpc/).
## Getting started
### Create a new VPC
1. Open the [Virtual Private Clouds tab](https://pipedream.com/settings/networks):
![]()
1. Click on **New VPC** in the upper right of the page:
![]()
2. Enter a network name and click **Create**:
![]()
3. It may take 5-10 minutes to complete setting up your network. The status will change to **Available** when complete:
![]()
### Run workflows within a VPC
To run workflows in a VPC, check the **Run in Private Network** option in workflow settings and select the network you created. All outbound network requests for the workflow will originate from the static egress IP for the VPC (both when testing a workflow or when running the workflow in production).
![]()
If you don’t see the network listed, the network setup may still be in progress. If the issue persists longer than 10 minutes, please [contact support](https://pipedream.com/support).
### Find the static outbound IP address for a VPC
You can view and copy the static outbound IP address for each VPC in your workspace from the [Virtual Private Cloud settings](https://pipedream.com/settings/networks). If you need to restrict access to sensitive resources (e.g., a database) by IP address, copy this address and configure it in your application with the `/32` CIDR block. Network requests from workflows running in the VPC will originate from this address.
![]()
## Managing a VPC
To rename or delete a VPC, navigate to the [Virtual Private Cloud settings](https://pipedream.com/settings/networks) for your workspace and select the option from the menu to the right of the VPC you want to manage.
## Self-hosting and VPC peering
If you’re interested in running Pipedream workflows in your own infrastructure, or configure VPC peering to allow Pipedream to communicate to resources in a private network, please reach out to our [Sales team](mailto:sales@pipedream.com).
## Limitations
* Only workflows can run in VPCs (other resources like sources or data stores are not currently supported). For example, [sources](/docs/workflows/building-workflows/triggers/) cannot yet run in VPCs.
* Creating a new network can take up to 5 minutes. Deploying your first workflow into a new network and testing that workflow for the first time can take up to 1 min. Subsequent operations should be as fast as normal.
* VPCs only provide static IPs for outbound network requests. This feature does not provide a static IP for or otherwise restrict inbound requests.
* You can’t set a default network for all new workflows in a workspace or project (you must select the network every time you create a new workflow). Please [reach out](https://pipedream.com/support) if you’re interesting in imposing controls like this in your workspace.
* Workflows running in a VPC will still route specific requests routed through [the shared Pipedream network](/docs/workflows/data-management/destinations/http/#ip-addresses-for-pipedream-http-requests):
* [`$.send.http()`](/docs/workflows/data-management/destinations/http/) requests
* Async options requests (these are requests that are made to populate options in drop down menus for actions while a building a workflow — e.g., the option to “select a Google Sheet” when using the “add row to Google Sheets” action)
## FAQ
### Will HTTP requests sent from Node.js, Python and the HTTP request steps use the assigned static IP address?
Yes, all steps that send HTTP requests from a workflow assigned to a VPC will use that VPC’s IP address to send HTTP requests.
This will also include `axios`, `requests`, `fetch` or any HTTP client you prefer in your language of choice.
The only exception are requests sent by `$.send.http()` or the HTTP requests used to populate async options that power props like “Select a Google Sheet” or “Select a Slack channel”. These requests will route through the [standard set of Pipedream IP addresses.](/docs/privacy-and-security/#hosting-details)
### Can a single workflow live within multiple VPCs?
No, a VPC can contain many workflows, but a single workflow can only belong to one VPC.
### Can I modify my VPC’s IP address to another address?
No, IP addresses are assigned to VPCs for you, and they are not changeable.
### How much will VPCs cost?
VPCs are available on the **Business** plan. [Upgrade your plan here](https://pipedream.com/pricing).
# Managing workspaces
Source: https://pipedream.com/docs/workspaces
When you sign up for Pipedream, you’ll either create a new workspace or join an existing one if you signed up from an invitation.
You can create and join any number of workspaces. For example, you can create one to work alone and another to collaborate with your team. You can also start working alone, then easily add others into your existing workspace to work together on workflows you’ve already built out.
Once you’ve created a new workspace, you can invite your team to create and edit workflows together, and organize them within projects and folders.
## Creating a new workspace
To create a new workspace,
1. Open the dropdown menu in the top left of the Pipedream dashboard
2. Select **New workspace**
3. You’ll be prompted to name the workspace (you can [change the name later](/docs/workspaces/#renaming-a-workspace))
## Workspace settings
Find your current [workspace settings](https://pipedream.com/settings/account) like current members, under the **Settings** navigation menu item on the left hand side. This is where you can manage your workspace settings, including the workspace name, members, and member permissions.
### Inviting others to a join a workspace
After opening your workspace settings, open the [Membership](https://pipedream.com/settings/users) tab.
* Invite people to your workspace by entering their email address and then clicking **Send**
* Or create an invite link to more easily share with a larger group (you can limit access to only specific email domains)
![]()
### Managing member permissions
Pipedream workspaces have three roles: **Owner**, **Admin**, and **Member**. Each role has different levels of access to workspace settings and resources.
#### Role overview
| Role | Description |
| ---------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| **Owner** | Full control over the workspace, including the ability to delete the workspace and manage other owners. A workspace can have multiple owners. |
| **Admin** | Can manage workspace settings, members, and security configurations, but cannot delete the workspace or manage owners. |
| **Member** | Basic access to view workspace details and work with resources based on project-level permissions. This is the default role for new members. |
#### Permissions by role
| Action | Owner | Admin | Member |
| ----------------------------------------------- | :---: | :---: | :----: |
| **Workspace management** | | | |
| Delete workspace | ✓ | | |
| Update workspace settings (name, notifications) | ✓ | ✓ | |
| View workspace details | ✓ | ✓ | ✓ |
| **User management** | | | |
| Invite users | ✓ | ✓ | ✓ |
| Remove non-owners (admins or users) | ✓ | ✓ | |
| Remove owners | ✓ | | |
| Grant or revoke admin role | ✓ | ✓ | |
| Grant or revoke owner role | ✓ | | |
| Leave workspace (remove self) | ✓ | ✓ | ✓ |
| **Security settings** | | | |
| Configure SSO | ✓ | ✓ | |
| Require MFA | ✓ | ✓ | |
Project-level permissions use a separate access control system with **Creator**, **Editor**, and **Viewer** roles. See [Project access controls](/docs/projects/access-controls) for details.
#### Promoting a member to admin
To promote a member to an admin level account in your workspace, click the 3 dots to the right of their email and select “Make Admin”.
![]()
#### Demoting an admin to a member
To demote an admin back to a member, click the 3 dots to the right of their email address and select “Remove Admin”.
![]()
### Finding your workspace’s ID
Visit your [workspace settings](https://pipedream.com/settings/account) and scroll down to the **API** section. You’ll see your workspace ID here.
### Requiring Two-Factor Authentication
As a workspace admin or owner on the [Business plan](https://pipedream.com/pricing), you’re able to **require** that all members in your workspace must enable 2FA on their account.
1. Open the Authentication tab in your [workspace settings](https://pipedream.com/settings/authentication) (you must be an admin or owner to make changes here)
2. Make sure you’re in the [correct workspace](/docs/workspaces/#switching-between-workspaces)
3. Click the toggle under **Require 2FA** — this will open a confirmation modal with some additional information
4. Once you enable the change in the modal, **all workspace members (including admins and owners) will immediately be required to configure 2FA on their account**. All new and existing workspace members will be required to set up 2FA the next time they sign in.
![]()
Anyone who is currently logged in to Pipedream will be temporarily signed out until they set up 2FA
If anyone is actively making changes to a workflow, their session may be interrupted. We recommend enabling the 2FA requirement in off hours.
### Configuring Single Sign-On (SSO)
Workspaces on the Business plan can configure Single Sign-On, so your users can login to Pipedream using your identity provider.
Pipedream supports SSO with Google, Okta, and any provider that supports the SAML protocol. See the guides below to configure SSO for your identity provider:
* [Okta](/docs/workspaces/sso/okta/)
* [Google](/docs/workspaces/sso/google/)
* [Other SAML provider](/docs/workspaces/sso/saml/)
### SCIM
Pipedream supports provisioning user accounts from your IdP via SCIM. Any workspace on the Business plan can configure Single Sign-On with SCIM.
### Renaming a workspace
To rename a workspace, open your [workspace settings](https://pipedream.com/settings/account) and navigate to the **General** tab.
Click the save button to save the changes.
Only workspace **owners** and **admins** and rename a workspace.
### Deleting a workspace
To delete a workspace, open your workspace settings and navigate to the **Danger Zone**.
Click the **Delete workspace** button and confirm the action by entering in your workspace name and `delete my workspace` into the text prompt.
Only workspace **owners** can delete a workspace.
Deleting a workspace will delete all **sources**, **workflows**, and other resources in your workspace.
Deleting a workspace is **irreversible** and permanent.
## Switching between workspaces
To switch between workspaces, open the dropdown menu in the top left of the Pipedream dashboard.
Select which workspace you’d like to start working within, and your Pipedream dashboard context will change to that workspace.
# Domain Verification
Source: https://pipedream.com/docs/workspaces/domain-verification
Pipedream requires that you verify ownership of your email domain in order to [configure SAML SSO](/docs/workspaces/sso/) for your workspace. If your email is `foo@example.com`, you need to verify ownership of `example.com`. If configuring Google OAuth (not SAML), you can disregard this section.
## Getting started
1. Navigate to the **[Verified Domains](https://pipedream.com/settings/domains)** section of your workspace settings
2. Enter the domain you’d like to use then click **Add Domain**
3. You’ll see a modal with instructions for adding a `TXT` record in the DNS configuration for your domain
4. DNS changes may take between a few minutes and up to 72 hours to propagate. Once they’re live, click the **Verify** button for the domain you’ve entered
5. Once Pipedream verifies the `TXT` record, we’ll show a green checkmark on the domain
Make sure to verify all your domains. There’s no limit on the number of domains you can verify for SSO, so if you use `example.com`, `example.net`, and `foo.example.com`, make sure to verify each one.
![]()
# Single Sign On Overview
Source: https://pipedream.com/docs/workspaces/sso
Pipedream supports Single Sign-On (SSO) with [Okta](/docs/workspaces/sso/okta/), [Google](/docs/workspaces/sso/google/), or [any provider](/docs/workspaces/sso/saml/) that supports SAML or Google OAuth, which allows IT and workspace administrators easier controls to manage access and security.
Using SSO with your Identity Provider (IdP) centralizes user login management and provides a single point of control for IT teams and employees.
## Requirements for SSO
* Your workspace must be on a [Business plan](https://pipedream.com/pricing)
* If using SAML, your Identity Provider must support SAML 2.0
* Only workspace admins and owners can configure SSO
* Your workspace admin or owner must [verify ownership](/docs/workspaces/sso/#verifying-your-email-domain) of the SSO email domain
The below content is for workspace admins and owners. Only workspace admins and owners have access to add verified domains, set up SSO, and configure workspace login methods.
## Verifying your Email Domain
In order to configure SAML SSO for your workspace, you first need to verify ownership of the email domain. If configuring Google OAuth (not SAML), you can skip this section.
[Refer to the guide here](/docs/workspaces/domain-verification/) to verify your email domain.
## Setting up SSO
Navigate to the [Authentication section](https://pipedream.com/settings/domains) in your workspace settings to get started.
### SAML SSO
1. First, make sure you’ve verified the domain(s) you intend to use for SSO ([see above](/docs/workspaces/sso/#verifying-your-email-domain))
2. Click the **Enable SSO** toggle and select **SAML**
3. If setting up SAML SSO, you’ll need to enter a metadata URL, which contains all the necessary configuration for Pipedream. Refer to the provider-specific docs for the detailed walk-through ([Okta](/docs/workspaces/sso/okta/), [Google Workspace](/docs/workspaces/sso/google/), [any other SAML provider](/docs/workspaces/sso/saml/)).
4. Click **Save**
### Google OAuth
1. Click the **Enable SSO** toggle and select **Google**
2. Enter the domain that you use with Google OAuth. For example, `vandalayindustries.com`
3. Click **Save**
## Restricting Login Methods
Once you’ve configured SSO for your workspace, you can restrict the allowed login methods for [non-workspace owners](/docs/workspaces/sso/#workspace-owners-can-always-sign-in-using-any-login-method).
![]()
| Login Method | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Any login method** | Everyone in the workspace can sign in either using SSO or via the login method they used to create their account (email and password, Google OAuth, GitHub) |
| **SSO only** | Workspace members and admins must [sign in using SSO](https://pipedream.com/auth/sso) |
| **SSO with guests** | When signing in using a verified email domain, members and admins must [sign in using SSO](https://pipedream.com/auth/sso). If signing in with a different domain (`gmail.com` for example), members (guests) can sign in using any login method. |
### Workspace owners can always sign in using any login method
In order to ensure you don’t get locked out of your Pipedream workspace in the event of an outage with your identity provider, workspace owners can always sign in via the login method they used to create the account (email and password, Google, or GitHub).
### Login methods are enforced when signing in to pipedream.com
This means if you are a member of 2 workspaces and one of them allows **any login method** but the other **requires SSO**, you will be required to sign in to Pipedream using SSO every time, independent of the workspace you are trying to access.
# Configure SSO with Google Workspace
Source: https://pipedream.com/docs/workspaces/sso/google
Pipedream supports Single Sign-On (SSO) with Google Workspace. This guide shows you how to configure SSO in Pipedream to authenticate with your Google org.
## Requirements
* SSO is only supported for [workspaces](/docs/workspaces/) on the Business plan. Visit the [Pipedream pricing page](https://pipedream.com/pricing) to upgrade.
* You need an administrator of your Pipedream workspace and someone who can [create SAML apps in Google Workspace](https://apps.google.com/supportwidget/articlehome?hl=en\&article_url=https%3A%2F%2Fsupport.google.com%2Fa%2Fanswer%2F6087519%3Fhl%3Den\&assistant_id=generic-unu\&product_context=6087519\&product_name=UnuFlow\&trigger_context=a) to configure SSO.
## Configuration
To configure SSO in Pipedream, you need to set up a [SAML application](https://apps.google.com/supportwidget/articlehome?hl=en\&article_url=https%3A%2F%2Fsupport.google.com%2Fa%2Fanswer%2F6087519%3Fhl%3Den\&assistant_id=generic-unu\&product_context=6087519\&product_name=UnuFlow\&trigger_context=a) in Google Workspace. If you’re a Google Workspace admin, you’re all set. Otherwise, coordinate with a Google Workspace admin before you continue.
In your **Google Workspace** admin console, select **Apps** > **Web and Mobile apps**
![]()
In the **Add app** menu, select the option to **Add custom SAML app**:
![]()
First, add **Pipedream** as the app name, and an app description that makes sense for your organization:
![]()
![]()
In the **Service provider details**, provide the following values:
* **ACS URL** — `https://api.pipedream.com/auth/saml/consume`
* **Entity ID** — Pipedream
* **Start URL** — `https://api.pipedream.com/auth/saml/`
replacing `` with the workspace name at [https://pipedream.com/settings/account](https://pipedream.com/settings/account). For example, if your workspace name is `example-workspace`, your start URL will be `https://api.pipedream.com/auth/saml/example-workspace`.
![]()
In the **Name ID** section, provide these values:
* **Name ID format** — `EMAIL`
* **Name ID** — Basic Information > Primary email
then press **Continue**.
![]()
Once the app is configured, visit the **User access** section to add Google Workspace users to your Pipedream SAML app. See [step 14 of the Google Workspace SAML docs](https://apps.google.com/supportwidget/articlehome?hl=en\&article_url=https%3A%2F%2Fsupport.google.com%2Fa%2Fanswer%2F6087519%3Fhl%3Den\&assistant_id=generic-unu\&product_context=6087519\&product_name=UnuFlow\&trigger_context=a) for more detail.
Pipedream requires access to SAML metadata at a publicly-accessible URL. This communicates public metadata about the identity provider (Google Workspace) that Pipedream can use to configure the SAML setup in Pipedream.
First, click the **Download Metadata** button on the left of the app configuration page:
![]()
**Host this file on a public web server where Pipedream can access it via URL**, for example: `https://example.com/metadata.xml`. You’ll use that URL in the next step.
In Pipedream, visit your workspace’s [authentication settings](https://pipedream.com/settings/authentication).
In the **Single Sign-On** section, select **SAML**, and add the URL from step 7 above in the **Metadata URL** field, then click Save.
![]()
Any user in your workspace can now log into Pipedream at [https://pipedream.com/auth/sso](https://pipedream.com/auth/sso) by entering your workspaces’s name (found in your [Settings](https://pipedream.com/settings/account)). You can also access your SSO sign in URL directly by visiting [https://pipedream.com/auth/org/your-workspace-name](https://pipedream.com/auth/org), where `your-workspace-name` is the name of your workspace.
## Important details
Before you configure the application in Google, make sure all your users have matching email addresses for their Pipedream user profile and their Google Workspace profile. Once SSO is enabled, they will not be able to change their Pipedream email address.
If a user’s Pipedream email does not match the email in their Google profile, they will not be able to log in.
If existing users signed up for Pipedream using an email and password, they will no longer be able to do so. They will only be able to sign in using SSO.
# Configure SSO with Okta
Source: https://pipedream.com/docs/workspaces/sso/okta
Pipedream supports Single Sign-On (SSO) with Okta. This guide shows you how to configure SSO in Pipedream to authenticate with your Okta org.
## Requirements
* SSO is only supported for [workspaces](/docs/workspaces/) on the Business plan. Visit the [Pipedream pricing page](https://pipedream.com/pricing) to upgrade.
* You must be an administrator of your Pipedream workspace
* You must have an Okta account
## Configuration
In your Okta **Admin** dashboard, select the **Applications** section and click **Applications** below that:
![]()
Click **Browse App Catalog**:
![]()
Search for “Pipedream” and select the Pipedream app:
![]()
![]()
Fill out the **General Settings** that Okta presents, and click **Done**:
![]()
Select the **Sign On** tab, and click **Edit** at the top right:
![]()
Scroll down to the **SAML 2.0** settings. In the **Default Relay State** section, enter `organization_username`:
![]()
Set any other configuration options you need in that section or in the **Credentials Details** section, and click **Save**.
In the **Sign On** section, you’ll see a section that includes the setup instructions for SAML:
![]()
Click the **Identity Provider metadata** link and copy the URL from your browser’s address bar:
![]()
Visit your [Pipedream workspace authentication settings](https://pipedream.com/settings/authentication). Click the toggle to **Enable SSO**, then click **Edit SSO Configuration**, and add the metadata URL in the **SAML** section and click **Save**:
![]()
Back in Okta, click on the **Assignments** tab of the Pipedream application. Click on the **Assign** dropdown and select **Assign to People**:
![]()
Assign the application to the relevant users in Okta, and Pipedream will configure the associated accounts on our end.
Users configured in your Okta app can log into Pipedream at [https://pipedream.com/auth/sso](https://pipedream.com/auth/sso) by entering your workspaces’s name (found in your [Settings](https://pipedream.com/settings/account)). You can also access your SSO sign in URL directly by visiting [https://pipedream.com/auth/org/your-workspace-name](https://pipedream.com/auth/org), where `your-workspace-name` is the name of your workspace.
## Important details
Before you configure the application in Okta, make sure all your users have matching email addresses for their Pipedream user profile and their Okta profile. Once SSO is enabled, they will not be able to change their Pipedream email address.
If a user’s Pipedream email does not match the email in their IDP profile, they will not be able to log in.
If existing users signed up for Pipedream using an email and password, they will no longer be able to do so. They will only be able to sign in using SSO.
# Configure SSO with Another SAML Provider
Source: https://pipedream.com/docs/workspaces/sso/saml
Pipedream supports Single Sign-On (SSO) with any identity provider that supports SAML 2.0. This guide shows you how to configure SSO in Pipedream to authenticate with your SAML provider.
If you use [Okta](/docs/workspaces/sso/okta/) or [Google Workspace](/docs/workspaces/sso/google/), please review the guides for those apps.
## Requirements
* SSO is only supported for [workspaces](/docs/workspaces/) on the Business plan. Visit the [Pipedream pricing page](https://pipedream.com/pricing) to upgrade.
* You need an administrator of your Pipedream workspace and someone who can create SAML apps in your identity provider to configure SSO.
## SAML metadata
| Name | Other names | Value |
| --------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| SP Entity ID | Audience, Audience Restriction, SP URL | `Pipedream` |
| SP Assertion Consumer Service (ACS) URL | Reply or destination URL | `https://api.pipedream.com/auth/saml/consume` |
| SP Single Sign-on URL | Start URL | `https://api.pipedream.com/auth/saml/`
replacing `` with the workspace name at [https://pipedream.com/settings/account](https://pipedream.com/settings/account). For example, if your workspace name is `example-workspace`, your start URL will be `https://api.pipedream.com/auth/saml/example-workspace`. |
## SAML attributes
* `NameID` —email
## Providing SAML metadata to Pipedream
Pipedream requires access to SAML metadata at a publicly-accessible URL. This communicates public metadata about the identity provider (your SSO provider) that Pipedream can use to configure the SAML setup in Pipedream.
Most SSO providers will provide a publicly-accessible metadata URL. If not, they should provide a mechanism to download the SAML metadata XML file. **Once you’ve configured your SAML app using the settings above, host this file on a public web server where Pipedream can access it via URL**, for example: `https://example.com/metadata.xml`.
Once you have a publicly-accessible URL that hosts your SAML metadata, visit your workspace’s [authentication settings](https://pipedream.com/settings/authentication) in Pipedream. In the **Single Sign-On** section, select **SAML**, and add your metadata URL to the **Metadata URL** field, then click **Save**.
![]()
Any user in your workspace can now log into Pipedream at [https://pipedream.com/auth/sso](https://pipedream.com/auth/sso) by entering your workspaces’s name (found in your [Settings](https://pipedream.com/settings/account)). You can also access your SSO sign in URL directly by visiting [https://pipedream.com/auth/org/your-workspace-name](https://pipedream.com/auth/org), where `your-workspace-name` is the name of your workspace.
## Important details
Before you configure the application in your IdP, make sure all your users have matching email addresses for their Pipedream user profile and their IdP profile. Once SSO is enabled, they will not be able to change their Pipedream email address.
If a user’s Pipedream email does not match the email in their IdP profile, they will not be able to log in.
If existing users signed up for Pipedream using an email and password, they will no longer be able to do so. They will only be able to sign in using SSO.