Getting started creating a new event source (Reddit)

This topic was automatically generated from Slack. You can find the original thread here.

Dylan Sather (Pipedream) : re: Reddit, go ahead and start on the New Link in a Subreddit source. That source should accept the subreddit name as a prop, similar to the action.

Here’s how I’d get started:

  1. Fork the PipedreamHQ/pipedream repo
  2. In the components directory of your fork, create a new reddit directory.
  3. This is a simple example of the structure of a directory for an app. First, you’ll need to create an “app file” - - which contains basic metadata about the app and shared props and methods. Here, the app name should be reddit (app: "reddit"). The propDefinitions section should contain props that are shared across multiple sources. Adding them in the app file allows you to reuse the props across sources. So when you add the subreddit prop, I’d recommend keeping it here. The methods section contains shared methods that can also be used across sources, as well.
  4. A note on auth. In workflows, the user’s connected account details for Reddit are accessible at auths.reddit. In sources, that auth data is exposed at this.$auth. So to reference the OAuth access token, for example, you’d use this.$auth.oauth_access_token.
  5. When you create a specific source, create a sources subdirectory within the reddit dir, then another subdir for the source name, then finally the source file itself. For example, this first source should live at components/reddit/sources/new-link-in-subreddit/new-link-in-subreddit.js.
  6. The structure of the source itself is defined by the component API. Here’s one example source, but you’ll find many more in the components dir under the relevant apps, so I’d recommend taking a look there for examples.
  7. Reddit doesn’t support webhooks, so you’ll need to poll the API endpoint on a schedule, looking for new items added since the last new post that was returned from the API. Luckily, this endpoint supports an after parameter that should allow you to fetch new items since a specific item ID. A lot of our sources operate in the same way: 1) pull first set of items, 2) store the last item / last time the source ran in this.db (see docs), 3) the next time the source runs, pull the last item ID from this.db and fetch new items added since then. Take a look at similar sources here, here, and here.
    Let me know if that helps get you started, or if you have any other questions.

SERW2039 : Thanks so much for this get started guide, . It totally helps me to get started. I am on the reddit source. About 7), I noticed the need for polling the API for new items. Yeah, I may get back with other questions.

SERW2039 : So I review all the documents, and all info is sinking in. Have one question regarding relationship between the app file and the source file. I notice many of the API requests needed by the source are defined as method in the app file. Does it means that most of the times the app file will have code accessing the app’s API to get data, then then it is called, consumed in the source file to emit events? I.e. the app file could have a method to pull a list, and then the source file could include logic to filter out the newest, or the newest and an arbitrary criteria (i.e. list item with a lenght meeting a minimal num of characters). The app file also has some common methods mostly defined in all apps, such as _getBaseUrl, or _getHeaders

Dylan Sather (Pipedream) : Yes typically we keep most of the API requests in the app file. Then the logic for how that API request is used, and how we handle the associated records, will typically be contained within the source itself.

SERW2039 : I also have a gap when it comes to the app file. How is that going to be deployed? Will that happen when I get a Pull Request accepted/merged on master with my app file (I.e and it is built by the PD’s CI/CD process? And then I will have my new-link-in-subreddit.js file be deployed with pd deploy or is it going to be deployed as well bia a probable PR? (and hence be “official”, that is, offitially available as a source for others PD users, with the github example from the file being source that used the existing and a “custom” source that is available only for the user pd deploying??

Dylan Sather (Pipedream) : Yes all great questions. When you have a PR ready I’ll show you more. For now, you’ll want to use pd dev to develop like you did in the quickstart. When we merge your PR to master, a Github action publishes the app file + component to our registry, which makes it available to all users.

SERW2039 : Hey Dylan, I don’t have a PR ready yet. But I have two blockers. One is that in a source using the requested new link source, the $auth object is null when deployed and thus reddit api calls fail, but in another source using reddit, the $auth is defined and reddit api work. the other blocker is i defined a with a simpler reddit api call in the source file, when i do pd dev, it fails saying of a 500 error (my guess is that the needs to be deployed throug the PR)

SERW2039 : I see you are OOO, so when you can we can discuss,

SERW2039 : hi , you there?

Dylan Sather (Pipedream) : Hi , sorry for the delay. Can you create a new PR with your current code so I can take a look?

SERW2039 : Yep, i have issued a PR now. Should show in the PipedreamHQ master.

SERW2039 : repo

SERW2039 : ah ok, looks like it’s been taken care by a process on vercel

Dylan Sather (Pipedream) : I need to turn the Vercel build off, that runs for our docs/ directory but is triggered on every PR (regardless of whether it touches docs). You can ignore the Vercel stuff for now

Dylan Sather (Pipedream) : taking a look at the PR now

Dylan Sather (Pipedream) : I added a couple of comments based on issues I observed that I think may have caused the issues you were seeing. Let me know if that helps

SERW2039 : ok, i’ll take a look, make changes accordingly and let you know, brb

SERW2039 : i have the changes and have comitted. should I issue a new PR?

Dylan Sather (Pipedream) : Did that solve the issues you were seeing?

Dylan Sather (Pipedream) : If not can you copy and paste or take a screenshot of the specific error you’re seeing?