Get all records with Data Stores API on New Stories by Keyword from Hacker News API

Pipedream makes it easy to connect APIs for Data Stores, Hacker News and 2,000+ other apps remarkably fast.

Trigger workflow on

New Stories by Keyword from the Hacker News API

Next, do this

Get all records with the Data Stores API

No credit card required

▶

Watch us build a workflow

4 min

Watch now ➜

Trusted by 800,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Hacker News trigger and Data Stores action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Stories by Keyword trigger
1. Connect your Hacker News account
2. Configure timer
3. Optional- Configure Keyword
Configure the Get all records action
1. Connect your Data Stores account
2. Configure Data Store
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Stories by Keyword on Hacker News

Description:Emit new stories that mention a specific keyword

Version:0.0.4

Key:hacker_news-new-stories-by-keyword

View on GitHub

Hacker News Overview#

The Hacker News API on Pipedream allows you to tap into the vibrant pool of stories, comments, and user data from one of tech's favorite forums. By leveraging this API, you can automate the process of extracting data for analysis, tracking mentions of specific topics, or even monitoring the performance of your own submissions. The real power lies in integrating this wealth of information with other services to create custom, automated workflows that save time and keep you informed.

Trigger Code#

const Parser = require("rss-parser");
const hacker_news = require("../../hacker_news.app.js");
const { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } = require("@pipedream/platform");

const parser = new Parser();

module.exports = {
  key: "hacker_news-new-stories-by-keyword",
  name: "New Stories by Keyword",
  description: "Emit new stories that mention a specific keyword",
  version: "0.0.4",
  type: "source",
  props: {
    hacker_news,
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    db: "$.service.db",
    // If you want a single search feed but multiple keywords, separate the keywords with " OR ":
    keyword: {
      type: "string",
      label: "Keyword",
      description: "Keyword to watch. Matches story titles. Leave blank to get all stories.",
      optional: true,
      default: "",
    },
  },
  methods: {
    // in theory if alternate setting title and description or aren't unique this won't work
    itemKey(item) {
      return item.guid || item.id || item.title || item.description;
    },
  },
  async run() {
    const seenKeys = this.db.get("seenKeys") || [];
    const seenKeysMap = seenKeys.reduce((acc, v) => {
      acc[v] = true;
      return acc;
    }, {});
    // All elements of an item are optional,
    // however at least one of title or description must be present.
    // should be listed from most recent to least recent
    const feed = await parser.parseURL(`https://hnrss.org/newest?q=${encodeURIComponent(this.keyword)}`);
    for (let idx = feed.items.length - 1; idx >= 0; idx--) {
      const item = feed.items[idx];
      const key = this.itemKey(item);
      // XXX throw if !key?
      if (seenKeysMap[key]) continue;
      seenKeys.unshift(key);
      seenKeysMap[key] = true; // just in case of dupes
      this.$emit(item, {
        id: this.itemKey(item),
        summary: item.title,
        ts: item.pubdate && +new Date(item.pubdate),
      });
    }
    if (seenKeys.length) {
      // XXX restrict by byte size instead of el size
      this.db.set("seenKeys", seenKeys.slice(0, 1000));
    }
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Hacker News	`hacker_news`	`app`	This component uses the Hacker News app.
	`timer`	`$.interface.timer`
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
Keyword	`keyword`	`string`	Keyword to watch. Matches story titles. Leave blank to get all stories.

Trigger Authentication#

The Hacker News API does not require authentication.

About Hacker News#

Social news website

Action#

Get all records on Data Stores

Description:Get all records in your [Pipedream Data Store](https://pipedream.com/data-stores/). The memory consumption of the workflow can be affected, since this action will be exposing, to the workflow, the entire data from the selected datastore.

Version:0.0.2

Key:data_stores-get-all-records

View on GitHub

Data Stores Overview#

Data Stores are a key-value store that allow you to persist state and share data across workflows. You can perform CRUD operations, enabling dynamic data management within your serverless architecture. Use it to save results from API calls, user inputs, or interim data; then read, update, or enrich this data in subsequent steps or workflows. Data Stores simplify stateful logic and cross-workflow communication, making them ideal for tracking process statuses, aggregating metrics, or serving as a simple configuration store.

Action Code#

import app from "../../data_stores.app.mjs";

export default {
  key: "data_stores-get-all-records",
  name: "Get all records",
  description: "Get all records in your [Pipedream Data Store](https://pipedream.com/data-stores/). The memory consumption of the workflow can be affected, since this action will be exposing, to the workflow, the entire data from the selected datastore.",
  version: "0.0.2",
  type: "action",
  props: {
    app,
    dataStore: {
      propDefinition: [
        app,
        "dataStore",
      ],
    },
  },
  async run({ $ }) {
    const { dataStore } = this;
    const keys = await dataStore.keys();
    const promises = [];
    for (const key of keys) {
      promises.push(dataStore.get(key));
    }
    const arrData = await Promise.all(promises);
    const objData = {};
    for (let i = 0; i < keys.length; i++) {
      objData[keys[i]] = arrData[i];
    }
    $.export("$summary", `Successfully returned ${keys.length} keys.`);
    return objData;
  },
};