components/foo
for example):
import
the package. You can’t use require
.
You also cannot mix ESM with CJS. This will not work:
.mjs
file extension for any components written as ES modules.
You’ll notice that many of the existing components are written as CommonJS modules. Please fix these and submit a pull request as you refactor related code. For example, if you’re developing new Spotify actions, and you notice the existing event sources use CommonJS, change them to ESM:
.js
to .mjs
using git mv
(e.g. git mv source.js source.mjs
).require
statements to import
s.module.exports
to export default
.key
and version
, and a friendly name
and description
. Action components require a type
field to be set to action
(sources will require a type to be set in the future). Action components require the description to include a link to the relevant documentation in the following format: [See the documentation](https://public-api.com)
key
must be unique across registry components and should follow the pattern:
app_name_slug
-slugified-component-name
Source keys should use past tense verbs that describe the event that occurred (e.g., linear_app-issue-created-instant
). For action keys, use active verbs to describe the action that will occur, (e.g., linear_app-create-issue
).
0.0.1
.
Pipedream registry components try to follow semantic versioning. From their site:
Given a version number MAJOR.MINOR.PATCH
, increment the:
MAJOR
version when you make incompatible API changes,MINOR
version when you add functionality in a backwards compatible manner, andPATCH
version when you make backwards compatible bug fixes.0.0.1
. If the action was at version 0.1.0
and you’ve fixed a bug, change it to 0.1.1
when committing your final code.
If you update a file, you must increment the versions of all components that import or are affected by the updated file.
components
directory of the pipedreamhq/pipedream
repo.
/components/[app_slug]/[app_slug].app.mjs
)/sources
and /actions
subfoldersjs
file equivalent to the slugified component name). For example, the path for the “Search Mentions” source for Twitter is /components/twitter/sources/search-mentions/search-mentions.mjs
.app_slug
, words in folder and file names are separated by dashes (-) (i.e., in kebab case)common.mjs
, utils.mjs
) must be placed within a common folder: /common/common.mjs
.package.json
package.json
in its root folder. If one doesn’t exist, run npm init
in the app’s root folder and customize the file using this package.json
as a template.
Each time you change the code for an app file, or change the dependencies for any app component, modify the package version
.
Save any dependencies in the component app directory:
throw
your own custom error that wraps the error from the API / lib. See the Airtable components for an example of custom error-handling and input validation.
In general, imagine you are a user troubleshooting an issue. Is the error easy-to-understand? If not, throw
a better error.
README
filesREADME.md
files within the same directory to describe how to use the action or source to users.
Here’s an example README.md
structure:
README.md
within the discord
component on the Pipedream registry. That same content is rendered within the Pipedream integration page for the Discord app.
You can add additional subheadings to each of the top level Overview
, Example Use Cases
, Getting Started
and Troubleshooting
headings:
Overview
, Getting Starting
and Troubleshooting
will appear within the corresponding App Marketplace page. All other headings will be ignored.scanDeltaItems
generator method in the example above follows this pattern.processEvent
method in this component for an example.[app_name_slug].app.mjs
. If an app file does not exist for your app, please reach out.
[foo=bar]
). This data will both help with reusability and will be surfaced in documentation for apps in the Pipedream marketplace. For example:
common
module to abstract elements that are used across to multiple components. The trade-off with this approach is that it increases complexity for end-users who have the option of customizing the code for components within Pipedream. When using this approach, the general pattern is:
.app.mjs
module contains the logic related to making the actual API calls (e.g. calling axios.get
, encapsulate the API URL and token, etc).common.mjs
module contains logic and structure that is not specific to any single component. Its structure is equivalent to a component, except that it doesn’t define attributes such as version
, dedupe
, key
, name
, etc (those are specific to each component). It defines the main logic/flow and relies on calling its methods (which might not be implemented by this component) to get any necessary data that it needs. In OOP terms, it would be the equivalent of a base abstract class.common.mjs
component by setting additional attributes (e.g. name
, description
, key
, etc) and potentially redefining any inherited methods.common.mjs
, utils.mjs
) must be placed within a common folder: /common/common.mjs
.common
is just a convention and depending on each case it might make sense to name any common module differently. For example, the AWS sources contains a common
directory instead of a common.mjs
file, and the directory contains several modules that are shared between different event sources.
`
)image-20210326151557417
image-20210326151706682
image-20210326151930885
Prop | Recommended Prop Variable Name |
---|---|
$.interface.http | http |
$.interface.timer | timer |
$.service.db | db |
$.service.db
to avoid potential typos and leverage encapsulation (e.g., see the Search Mentions event source for Twitter).
unique
, greatest
, last
) vs developing custom deduping code. Develop custom deduping code if the existing strategies do not support the requirements for a source.
deploy()
hook during source creation (see the Jotform: New Submission source)steps.trigger.event
) and make sure to remove or scrub any sensitive or personal data (you can also copy this from the app’s API docs)test-event.mjs
in the same directory as the component source and export the JSON event via export default
(example)sampleEmit
(example)sampleEmit
object (example)generate-sample-event
$.service.db
whenever the API accepts a since_id
or “since timestamp” (or equivalent). Some apps (e.g., Github) do not count requests that do not return new results against a user’s API quota.
If the service has a well-supported Node.js client library, it’ll often build in retries for issues like rate limits, so using the client lib (when available) should be preferred. In the absence of that, Bottleneck can be useful for managing rate limits. 429s should be handled with exponential backoff (instead of just letting the error bubble up).
activate()
hook, and deleted when components are deactivated or deleted via the deactivate()
hook.
Description | Method Name |
---|---|
Method to create a webhook subscription | createHook() |
Method to delete a webhook subscription | deleteHook() |
$.service.db
for a source using the key hookId
. This ID will be referenced when managing or deleting the webhook. Note: some apps may not return a unique ID for the registered webhook (e.g., Jotform).
$.service.db
key, and use the saved value to validate incoming events.
@pipedream/platform
axios for all HTTP Requestsaxios
package doesn’t return useful debugging data to the user when it throw
s errors on HTTP 4XX and 5XX status codes. This makes it hard for the user to troubleshoot the issue.
Instead, use @pipedream/platform
axios.
return
data from an action, it’s exposed as a step export for users to reference in future steps of their workflow. Return JavaScript objects in all cases, unless there’s a specific reason not to.
For example, some APIs return XML responses. If you return XML from the step, it’s harder for users to parse and reference in future steps. Convert the XML to a JavaScript object, and return that, instead.
maxRequests
prop) to avoid exceeding Twitter’s rate limits. See the Airtable components for an example of using a maxRecords
prop to optionally limit the maximum number of records to return.
$.summary
to Summarize What Happened/tmp
directory and exporting a reference to the file.
@pipedream/platform
package.
sql
, and it uses (whenever possible) the underlying’s database schema information to provide auto-complete suggestions. Each database engine not only has its own SQL dialect, but also its own way of inspecting the schemas and table information it stores. For this reason, each app file must implement the logic that’s applicable to the target engine.
To support the schema retrieval, the app file must implement a method called getSchema
that takes no parameters, and returns a data structure with a format like this:
lib/sql-prop.ts
file in the @pipedream/platform
package define the schema format and the signature of the getSchema
method. You can also check out existing examples in the MySQL, PostgreSQL and Snowflake components.
lib/sql-proxy.ts
file in the @pipedream/platform
package. This interface defines the following methods:
getClientConfiguration
: This method takes no parameters and returns an object that can be fed directly to the database’s client library to initialize/establish a connection to the database. This guarantees that both the component and the proxy service use the same connection settings, so make sure the component uses this method when initializing the client.
executeQuery
: This method takes a query object and returns the result of executing the query against the database. The Pipedream runtime will replace this method with a call to the proxy service, so every component must make use of this method in order to support this feature.
proxyAdapter
: This method allows the proxy service to take the arguments passed to the executeQuery
method and transform them into a “generic” query object that the service can then use. The expected format looks something like this: