Append to record with Data Stores API on New Threat List Updated from Google Safe Browsing API

Pipedream makes it easy to connect APIs for Data Stores, Google Safe Browsing and 2,200+ other apps remarkably fast.

Trigger workflow on

New Threat List Updated from the Google Safe Browsing API

Next, do this

Append to record with the Data Stores API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Google Safe Browsing trigger and Data Stores action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Threat List Updated trigger
1. Connect your Google Safe Browsing account
2. Configure timer
3. Select one or more Threat Lists
Configure the Append to record action
1. Connect your Data Stores account
2. Configure Data Store
3. Select a Key
4. Configure Value
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Threat List Updated on Google Safe Browsing

Description:Emit new event when a threat list is updated. [See the documentation](https://developers.google.com/safe-browsing/v4/reference/rest/v4/threatListUpdates/fetch#ListUpdateRequest)

Version:0.0.1

Key:google_safebrowsing-threat-list-updated

View on GitHub

Google Safe Browsing Overview#

The Google Safe Browsing API lets you check URLs against Google's constantly updated lists of unsafe web resources. These include social engineering sites (like phishing and deceptive sites) and sites that host malware or unwanted software. Within Pipedream, you can leverage this API to automate the process of scanning URLs in various contexts, such as user-generated content, emails, or within applications that require high-security measures.

Trigger Code#

import safebrowsing from "../../google_safebrowsing.app.mjs";
import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";

export default {
  key: "google_safebrowsing-threat-list-updated",
  name: "New Threat List Updated",
  description: "Emit new event when a threat list is updated. [See the documentation](https://developers.google.com/safe-browsing/v4/reference/rest/v4/threatListUpdates/fetch#ListUpdateRequest)",
  version: "0.0.1",
  type: "source",
  dedupe: "unique",
  props: {
    safebrowsing,
    db: "$.service.db",
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    threatLists: {
      propDefinition: [
        safebrowsing,
        "threatLists",
      ],
    },
  },
  methods: {
    _getClientState() {
      return this.db.get("clientState");
    },
    _setClientState(clientState) {
      this.db.set("clientState", clientState);
    },
    generateMeta({
      threatType, platformType, threatEntryType, newClientState,
    }) {
      return {
        id: `${threatType}${platformType}${threatEntryType}${newClientState}`,
        summary: `${threatType} / ${platformType} / ${threatEntryType} Updated`,
        ts: Date.now(),
      };
    },
  },
  async run() {
    const threatLists = this.threatLists.map((list) => JSON.parse(list));
    const state = this._getClientState();

    const listUpdateRequests = state
      ? threatLists.map((list) => ({
        ...list,
        state,
      }))
      : threatLists;

    const { listUpdateResponses } = await this.safebrowsing.fetchThreatListUpdates({
      data: {
        listUpdateRequests,
      },
    });

    if (!listUpdateResponses?.length) {
      return;
    }

    for (const response of listUpdateResponses) {
      const meta = this.generateMeta(response);
      this.$emit(response, meta);
    }

    this._setClientState(listUpdateResponses[0].newClientState);
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Google Safe Browsing	`safebrowsing`	`app`	This component uses the Google Safe Browsing app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`
Threat Lists	`threatLists`	`string[]`	Select a value from the drop down menu.

Trigger Authentication#

Google Safe Browsing uses API keys for authentication. When you connect your Google Safe Browsing account, Pipedream securely stores the keys so you can easily authenticate to Google Safe Browsing APIs in both code and no-code steps.

To use the Google Safebrowsing API, you must Create a Project on Google Developer Console, then Setup an API Key, and activate Google Safebrowsing APIs (see Activate and Deactivate APIs)

About Google Safe Browsing#

Safe Browsing is a Google service that lets client applications check URLs against Google's constantly updated lists of unsafe web resources.

Action#

Append to record on Data Stores

Description:Append to a record in your data store [Pipedream Data Store](https://pipedream.com/data-stores/). If the record does not exist, a new record will be created in an array format.

Version:0.0.2

Key:data_stores-append-to-record

View on GitHub

Data Stores Overview#

Data Stores are a key-value store that allow you to persist state and share data across workflows. You can perform CRUD operations, enabling dynamic data management within your serverless architecture. Use it to save results from API calls, user inputs, or interim data; then read, update, or enrich this data in subsequent steps or workflows. Data Stores simplify stateful logic and cross-workflow communication, making them ideal for tracking process statuses, aggregating metrics, or serving as a simple configuration store.

Action Code#

import { ConfigurationError } from "@pipedream/platform";
import app from "../../data_stores.app.mjs";

export default {
  key: "data_stores-append-to-record",
  name: "Append to record",
  description: "Append to a record in your data store [Pipedream Data Store](https://pipedream.com/data-stores/). If the record does not exist, a new record will be created in an array format.",
  version: "0.0.2",
  type: "action",
  props: {
    app,
    dataStore: {
      propDefinition: [
        app,
        "dataStore",
      ],
    },
    key: {
      propDefinition: [
        app,
        "key",
        ({ dataStore }) => ({
          dataStore,
        }),
      ],
      description: "Enter a key for the record set you'd like to append to. Refer to your existing keys [here](https://pipedream.com/data-stores/).",
    },
    value: {
      propDefinition: [
        app,
        "value",
      ],
    },
  },
  async run({ $ }) {
    const {
      key,
      value,
    } = this;
    const currentValue = await this.dataStore.get(key);
    if (currentValue && !Array.isArray(currentValue)) {
      throw new ConfigurationError("The value for the specified key is not an array. You can only append records to an array.");
    }
    const recordSet = currentValue ?? [];
    const parsedValue = this.app.parseValue(value);
    recordSet.push(parsedValue);
    await this.dataStore.set(key, recordSet);
    // eslint-disable-next-line multiline-ternary
    $.export("$summary", `Successfully ${currentValue ? "appended to the record for" : "created new record with the"} key: \`${key}\`.`);
    return {
      key,
      value: parsedValue,
    };
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Data Stores	`app`	`app`	This component uses the Data Stores app.
Data Store	`dataStore`	`data_store`	Select an existing Data Store or create a new one.
Key	`key`	`string`	Select a value from the drop down menu.
Value	`value`	`any`	Enter a string, object, or array.