Event sources operate primarily as workflow triggers. When you add a new app-based trigger to your workflow, you're creating an event source.
Event sources run as their own resources, separate from workflows, for two reasons:
- A single event sources can trigger more than one workflow. If you have a data source that you want to run multiple workflows, you can create an event source once and use it as the trigger for each workflow.
- If you need to consume events emitted by event sources in your own application, you don't need to run a workflow: you can use Pipedream's REST API or a private, real-time SSE stream to access the event data directly.
You can view your event sources at https://pipedream.com/sources (opens new window). Here, you'll see the events a specific source has emitted, as well as the logs and configuration for that source.
- Creating event sources
- Consuming events from sources
- Example: HTTP source
- Example: Cron jobs
- Example: RSS
- Test Events
- Publishing a new event source, or modifying an existing source
Event sources collect data from apps or service like Github, Twitter, and Google Calendar, then emit this data as individual events. These events trigger linked workflows, and can be retrieved using the API or SSE interfaces.
If the service supports webhooks or another mechanism for real-time data delivery, the event source uses it. For example, Google Sheets supports webhooks, which allows Google Sheets event sources to emit updates instantly.
If a service doesn't support real-time event delivery, Pipedream polls the API for updates every few minutes, emitting events as the API produces them. For example, Airtable doesn't support webhooks, so we poll their API for new records added to a table.
# Creating event sources
You can create event sources from the Pipedream UI or CLI.
# Creating a source from the UI
Visit https://pipedream.com/sources (opens new window) and click the New + button at the top right to create a new event source. You'll see a list of sources tied to apps (like Twitter and Github) and generic interfaces (like HTTP). Select your source, and you'll be asked to connect any necessary accounts (for example, the Twitter source requires you authorize Pipedream access to your Twitter account), and enter the values for any configuration settings tied to the source.
# Creating a source from the CLI
Download the CLI and run:
This will bring up an interactive menu prompting you to select a source. Once selected, you'll be asked to connect any necessary accounts (for example, the Twitter source requires you authorize Pipedream access to your Twitter account), and enter the values for any configuration settings tied to the source.
# Consuming events from sources
You can view the events for a source in the sources UI, under the Events section of that source.
You can also trigger a Pipedream workflow every time your source emits a new event. This lets you run workflows for every new tweet, every new item in an RSS feed, and more.
Finally, you can consume events programmatically, outside the Pipedream platform, in a few different ways:
- In real-time, using the SSE stream linked to your source
- In real-time, via the CLI's
- In batch, using the REST API
# Example: HTTP source
The simplest event source is an HTTP source.
When you create an HTTP source:
- You get a unique HTTP endpoint that you can send any HTTP request to.
- You can view the details of any HTTP request sent to your endpoint: its payload, headers, and more.
- You can delete the source and its associated events once you're done.
HTTP sources are essentially request bins (opens new window) that can be managed via API.
HTTP sources are a good example of how you can turn an event stream into an API: the HTTP requests are the event stream, generated from your application, client browsers, webhooks, etc. Then, you can retrieve HTTP requests via Pipedream's REST API, or stream them directly to other apps using the SSE interface.
# Example: Cron jobs
You can also use event sources to run any Node code on a schedule, allowing you to poll a service or API for data and emit that data as an event. The emitted events can trigger Pipedream workflows, and can be retrieved using Pipedream's REST API or SSE interface.
# Example: RSS
You can run an event source that polls an RSS for new items and emits them in real time as formatted JSON.
# Test Events
Test events are critical for developing workflows effectively. Without a test event, you won't be able to test your workflow end to end in the workflow builder or see the shape of the event data that triggers the workflow.
Test events can be sourced from recent data, generated by real actions, or generated manually.
# Recent past events
During creation, most sources will attempt to fetch up to 50 of the most recent events.
Not all sources will fetch events automatically
Not all sources support pre-fetched events currently, but you can still generate test events yourself by performing the action that triggers the source.
# Generate a real event manually
If there aren't any existing events for the source to fetch, the next best way to produce a test event for your workflow is to generate one in the relevant application.
For example, if your workflow is triggered by New Files in Dropbox (opens new window), try adding a file to your Dropbox account.
# Generate a sample test event
Certain event sources support the ability to generate test events using static sample event data.
# Hardcoded sample events
For example, the New Message from Discord source (opens new window) references this static event (opens new window), which is a hardcoded file with purely representative data. The goal for this feature is to provide your workflow with the expected event shape to expect, but without the expected values.
# Customizing the sample event
If the provided static event is not sufficient for you to develop your workflow because you need to check against specific values (but are not able to generate real triggering events in the application), you can also customize the sample event.
- Select the arrow on the right side of the "Generate Test Event" button
- Select "Modify Test Event"
- Make whatever changes you'd like to the sample event
- Click "Send Custom Test Event"
To reset the custom event JSON back to the sample event, click the "Reset" button at the top right of the test event window:
Sample data may contain inconsistencies
Whenever possible, we recommend using a recent real event, either retrieved automatically or generated manually by an action.
Generating test data manually can result in typos or missing fields that your workflow depends on.
# Publishing a new event source, or modifying an existing source
Anyone can create an event source or edit an existing one. The code for all event sources is public, and kept in the
PipedreamHQ/pipedream repo (opens new window). Read this quickstart and see the event source API docs to learn more about the source development process.
Event sources are subject to the same limits as Pipedream workflows, except:
- Sources have a default timeout of 5 min.
- Memory is fixed at 256MB.