Create Droplet with Digital Ocean API on BigQuery - New Row from Google Cloud API

Pipedream makes it easy to connect APIs for Digital Ocean, Google Cloud and 2,000+ other apps remarkably fast.

Trigger workflow on

BigQuery - New Row from the Google Cloud API

Next, do this

Create Droplet with the Digital Ocean API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 800,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Google Cloud trigger and Digital Ocean action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the BigQuery - New Row trigger
1. Connect your Google Cloud account
2. Configure Polling interval
3. Configure Event Size
4. Select a Dataset ID
5. Select a Table Name
6. Select a Unique Key
Configure the Create Droplet action
1. Connect your Digital Ocean account
2. Configure Name
3. Select a Region
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

BigQuery - New Row on Google Cloud

Description:Emit new events when a new row is added to a table

Version:0.1.6

Key:google_cloud-bigquery-new-row

View on GitHub

Google Cloud Overview#

The Google Cloud API opens a world of possibilities for enhancing cloud operations and automating tasks. It empowers you to manage, scale, and fine-tune various services within the Google Cloud Platform (GCP) programmatically. With Pipedream, you can harness this power to create intricate workflows, trigger cloud functions based on events from other apps, manage resources, and analyze data, all in a serverless environment. The ability to interconnect GCP services with numerous other apps enriches automation, making it easier to synchronize data, streamline development workflows, and deploy applications efficiently.

Trigger Code#

import crypto from "crypto";
import { isString } from "lodash-es";
import googleCloud from "../../google_cloud.app.mjs";
import common from "../common/bigquery.mjs";

export default {
  ...common,
  key: "google_cloud-bigquery-new-row",
  // eslint-disable-next-line pipedream/source-name
  name: "BigQuery - New Row",
  description: "Emit new events when a new row is added to a table",
  version: "0.1.6",
  dedupe: "unique",
  type: "source",
  props: {
    ...common.props,
    tableId: {
      propDefinition: [
        googleCloud,
        "tableId",
        ({ datasetId }) => ({
          datasetId,
        }),
      ],
    },
    uniqueKey: {
      type: "string",
      label: "Unique Key",
      description: "The name of a column in the table to use for deduplication. See [the docs](https://github.com/PipedreamHQ/pipedream/tree/master/components/google_cloud/sources/bigquery-new-row#technical-details) for more info.",
      async options(context) {
        const { page } = context;
        if (page !== 0) {
          return [];
        }

        const columnNames = await this._getColumnNames();
        return columnNames.sort();
      },
    },
  },
  hooks: {
    ...common.hooks,
    async deploy() {
      await this._validateColumn(this.uniqueKey);
      const lastResultId = await this._getIdOfLastRow(this.getInitialEventCount());
      this._setLastResultId(lastResultId);
    },
    async activate() {
      if (this._getLastResultId()) {
        // ID of the last result has already been initialised during deploy(),
        // so we skip the rest of the activation.
        return;
      }

      await this._validateColumn(this.uniqueKey);
      const lastResultId = await this._getIdOfLastRow();
      this._setLastResultId(lastResultId);
    },
    deactivate() {
      this._setLastResultId(null);
    },
  },
  methods: {
    ...common.methods,
    _getLastResultId() {
      return this.db.get("lastResultId");
    },
    _setLastResultId(lastResultId) {
      this.db.set("lastResultId", lastResultId);
      console.log(`
        Next scan of table '${this.tableId}' will start at ${this.uniqueKey}=${lastResultId}
      `);
    },
    /**
     * Utility method to make sure that a certain column exists in the target
     * table. Useful for SQL query sanitizing.
     *
     * @param {string} columnNameToValidate The name of the column to validate
     * for existence
     */
    async _validateColumn(columnNameToValidate) {
      if (!isString(columnNameToValidate)) {
        throw new Error("columnNameToValidate must be a string");
      }

      const columnNames = await this._getColumnNames();
      if (!columnNames.includes(columnNameToValidate)) {
        throw new Error(`Nonexistent column: ${columnNameToValidate}`);
      }
    },
    async _getColumnNames() {
      const table = this.googleCloud
        .getBigQueryClient()
        .dataset(this.datasetId)
        .table(this.tableId);
      const [
        metadata,
      ] = await table.getMetadata();
      const { fields } = metadata.schema;
      return fields.map(({ name }) => name);
    },
    async _getIdOfLastRow(offset = 0) {
      const limit = offset + 1;
      const query = `
        SELECT *
        FROM \`${this.tableId}\`
        ORDER BY \`${this.uniqueKey}\` DESC
        LIMIT @limit
      `;
      const queryOpts = {
        query,
        params: {
          limit,
        },
      };
      const rows = await this.getRowsForQuery(queryOpts, this.datasetId);
      if (rows.length === 0) {
        console.log(`
          No records found in the target table, will start scanning from the beginning
        `);
        return;
      }

      const startingRow = rows.pop();
      return startingRow[this.uniqueKey];
    },
    getQueryOpts() {
      const lastResultId = this._getLastResultId();
      const query = `
        SELECT *
        FROM \`${this.tableId}\`
        WHERE \`${this.uniqueKey}\` >= @lastResultId
        ORDER BY \`${this.uniqueKey}\` ASC
      `;
      const params = {
        lastResultId,
      };
      return {
        query,
        params,
      };
    },
    generateMeta(row, ts) {
      const id = row[this.uniqueKey];
      const summary = `New row: ${id}`;
      return {
        id,
        summary,
        ts,
      };
    },
    generateMetaForCollection(rows, ts) {
      const hash = crypto.createHash("sha1");
      rows
        .map((i) => i[this.uniqueKey])
        .map((i) => i.toString())
        .forEach((i) => hash.update(i));
      const id = hash.digest("base64");

      const rowCount = rows.length;
      const entity = rowCount === 1
        ? "row"
        : "rows";
      const summary = `${rowCount} new ${entity}`;

      return {
        id,
        summary,
        ts,
      };
    },
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Google Cloud	`googleCloud`	`app`	This component uses the Google Cloud app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
Polling interval	`timer`	`$.interface.timer`	How often to run your query
Event Size	`eventSize`	`integer`	The number of rows to include in a single event (by default, emits 1 event per row)
Dataset ID	`datasetId`	`string`	Select a value from the drop down menu.
Table Name	`tableId`	`string`	Select a value from the drop down menu.
Unique Key	`uniqueKey`	`string`	Select a value from the drop down menu.

Trigger Authentication#

Google Cloud uses API keys for authentication. When you connect your Google Cloud account, Pipedream securely stores the keys so you can easily authenticate to Google Cloud APIs in both code and no-code steps.

Create a service account in GCP and set the permissions you need for Pipedream workflows.
Generate a service account key
Download the key details in JSON format
Open the JSON in a text editor, and copy and paste its contents here.

About Google Cloud#

The Google Cloud Platform, including BigQuery

Action#

Create Droplet on Digital Ocean

Description:Creates a droplet. [See the docs here](https://docs.digitalocean.com/reference/api/api-reference/#operation/create_droplet)

Version:0.1.2

Key:digital_ocean-create-droplet

View on GitHub

Digital Ocean Overview#

The Digital Ocean API provides programmatic access to manage Digital Ocean resources such as Droplets, Spaces, and Databases. With Pipedream, you can harness this API to automate a variety of tasks, like spinning up new servers, scaling resources, or integrating cloud infrastructure management into your workflow. It's a powerful tool for DevOps automation, allowing for the dynamic management of infrastructure in response to events or schedules.

Action Code#

import digitalOceanApp from "../../digital_ocean.app.mjs";

export default {
  key: "digital_ocean-create-droplet",
  name: "Create Droplet",
  description: "Creates a droplet. [See the docs here](https://docs.digitalocean.com/reference/api/api-reference/#operation/create_droplet)",
  version: "0.1.2",
  type: "action",
  props: {
    digitalOceanApp,
    name: {
      label: "Name",
      type: "string",
      description: "Human-readable string to use when displaying the Droplet name.",
    },
    region: {
      label: "Region",
      type: "string",
      description: "Unique slug identifier for the region to deploy this Droplet in.",
      reloadProps: true,
      async options() {
        return this.digitalOceanApp.fetchRegionsOpts();
      },
    },
  },
  async additionalProps() {
    if (!this.region) {
      return {};
    }
    return {
      image: {
        label: "Image",
        type: "string",
        description: "The image ID of a public or private image, or the unique slug identifier for a public image. This image will be the base image for this Droplet.",
        reloadProps: false,
        options: async () => {
          return this.digitalOceanApp.fetchImageOpts(this.region);
        },
      },
      size: {
        label: "Size",
        type: "string",
        description: "Unique slug identifier for the size to select for this Droplet.",
        reloadProps: false,
        options: async () => {
          return this.digitalOceanApp.fetchSizeOpts(this.region);
        },
      },
      volumes: {
        label: "Volumes",
        type: "string[]",
        description: "A flat array including the unique string identifier for each Block Storage volume to be attached to the Droplet. At the moment a volume can only be attached to a single Droplet.",
        optional: true,
        reloadProps: false,
        options: async () => {
          return this.digitalOceanApp.fetchVolumeOpts(this.region);
        },
      },
      sshKeys: {
        label: "SSH keys",
        type: "string[]",
        description: "An array containing the IDs or fingerprints of the SSH keys that you wish to embed in the Droplet's root account upon creation.",
        optional: true,
        reloadProps: false,
        options: async () => {
          return this.digitalOceanApp.fetchSshKeys();
        },
      },
      backups: {
        label: "Enable Automated Backup",
        type: "boolean",
        description: "A boolean indicating whether automated backups should be enabled for the Droplet. Automated backups can only be enabled when the Droplet is created.",
        optional: true,
      },
      ipv6: {
        label: "Enable IPv6",
        type: "boolean",
        description: "A boolean indicating whether IPv6 is enabled on the Droplet.",
        optional: true,
      },
      userData: {
        label: "User Data",
        type: "string",
        description: "A string containing 'user data' which may be used to configure the Droplet on first boot, often a 'cloud-config' file or Bash script. It must be plain text and may not exceed 64 KiB in size.",
        optional: true,
      },
      privateNetworking: {
        label: "Enable Private Networking",
        type: "boolean",
        description: "A boolean indicating whether private networking is enabled for the Droplet. Private networking is currently only available in certain regions.",
        optional: true,
      },
      monitoring: {
        label: "Enable Monitoring",
        type: "boolean",
        description: "A boolean indicating whether to install the DigitalOcean agent for monitoring.",
        optional: true,
      },
      tags: {
        label: "Tags",
        type: "string[]",
        description: "A flat array of tag names as strings to apply to the Droplet after it is created. Tag names can either be existing or new tags.",
        optional: true,
      },
    };
  },
  async run({ $ }) {
    const api = this.digitalOceanApp.digitalOceanWrapper();
    const newDropletData = {
      name: this.name,
      region: this.region,
      size: this.size,
      image: this.image,
      ssh_keys: this.sshKeys,
      backups: this.backups,
      ipv6: this.ipv6,
      user_data: this.userData,
      private_networking: this.privateNetworking,
      volumes: this.volumes,
      tags: this.tags,
      monitoring: this.monitoring,
    };
    const response = await api.droplets.create(newDropletData);
    $.export("$summary", `Successfully created droplet ${response.droplet.name}.`);
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Digital Ocean	`digitalOceanApp`	`app`	This component uses the Digital Ocean app.
Name	`name`	`string`	Human-readable string to use when displaying the Droplet name.
Region	`region`	`string`	Select a value from the drop down menu.

Action Authentication#

Digital Ocean uses OAuth authentication. When you connect your Digital Ocean account, Pipedream will open a popup window where you can sign into Digital Ocean and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Digital Ocean API.

Pipedream requests the following authorization scopes when you connect your account:

readwrite