Scrape URL List with Browserless API on New Comment (Instant) from Google Sheets API

Pipedream makes it easy to connect APIs for Browserless, Google Sheets and 2,400+ other apps remarkably fast.

Trigger workflow on

New Comment (Instant) from the Google Sheets API

Next, do this

Scrape URL List with the Browserless API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Google Sheets trigger and Browserless action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Comment (Instant) trigger
1. Connect your Google Sheets account
2. Configure Push notification renewal schedule
3. Select a Spreadsheet
Configure the Scrape URL List action
1. Connect your Browserless account
2. Configure URLs
3. Configure Selectors
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Comment (Instant) on Google Sheets

Description:Emit new event each time a comment is added to a spreadsheet.

Version:0.0.3

Key:google_sheets-new-comment

View on GitHub

Google Sheets Overview#

The Google Sheets API allows for the creation, reading, updating, and deletion of data within Google Sheets, enabling a robust platform for spreadsheet management and data manipulation. Through Pipedream, you can craft serverless workflows that respond to various triggers, such as webhook events, emails, or scheduled times, to interact with Google Sheets. This synergy can automate reporting, synchronize data across applications, manage inventory, track leads in a CRM, or even conduct survey analysis by updating and retrieving sheet data on the fly.

Trigger Code#

import httpBase from "../common/http-based/sheet.mjs";
import sampleEmit from "./test-event.mjs";

export default {
  ...httpBase,
  key: "google_sheets-new-comment",
  name: "New Comment (Instant)",
  description: "Emit new event each time a comment is added to a spreadsheet.",
  version: "0.0.3",
  dedupe: "unique",
  type: "source",
  methods: {
    ...httpBase.methods,
    _getLastTs() {
      return this.db.get("lastTs");
    },
    _setLastTs(lastTs) {
      this.db.set("lastTs", lastTs);
    },
    generateMeta(comment) {
      return {
        id: comment.id,
        summary: `New Comment: ${comment.content}`,
        ts: Date.parse(comment.createdTime),
      };
    },
    takeSheetSnapshot() {},
    getSheetId() {
      return this.sheetID.toString();
    },
    async processSpreadsheet() {
      const comments = [];
      const lastTs = this._getLastTs();
      const results = this.googleSheets.listComments(this.sheetID, lastTs);
      for await (const comment of results) {
        comments.push(comment);
      }
      if (!comments.length) {
        return;
      }
      this._setLastTs(comments[0].createdTime);
      comments.reverse().forEach((comment) => {
        const meta = this.generateMeta(comment);
        this.$emit(comment, meta);
      });
    },
  },
  async run(event) {
    if (event.timestamp) {
      // Component was invoked by timer
      return this.renewSubscription();
    }

    if (!this.isEventRelevant(event)) {
      console.log("Sync notification, exiting early");
      return;
    }

    await this.processSpreadsheet();
  },
  sampleEmit,
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Google Sheets	`googleSheets`	`app`	This component uses the Google Sheets app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
N/A	`http`	`$.interface.http`	This component uses `$.interface.http` to generate a unique URL when the component is first instantiated. Each request to the URL will trigger the `run()` method of the component.
Push notification renewal schedule	`timer`	`$.interface.timer`	The Google Drive API requires occasionally renewal of push notification subscriptions. This runs in the background, so you should not need to modify this schedule.
Spreadsheet	`sheetID`	`string`	Select a value from the drop down menu.

Trigger Authentication#

Google Sheets uses OAuth authentication. When you connect your Google Sheets account, Pipedream will open a popup window where you can sign into Google Sheets and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Google Sheets API.

Pipedream requests the following authorization scopes when you connect your account:

emailprofilehttps://www.googleapis.com/auth/drive

About Google Sheets#

Use Google Sheets to create and edit online spreadsheets. Get insights together with secure sharing in real-time and from any device.

Action#

Scrape URL List on Browserless

Description:Scrape content from a list of pages. [See the documentation](https://www.browserless.io/docs/scrape).

Version:0.0.2

Key:browserless-scrape-url-list

View on GitHub

Browserless Overview#

The Browserless API on Pipedream allows you to automate browser actions without the overhead of managing your own browser infrastructure. This service provides a way to run Chrome browser sessions programmatically, making it ideal for web scraping, automated testing, and screenshot capture. Leveraging this on Pipedream, you can create serverless workflows that interact with web pages, extract data, and perform actions as a human would, all in a scalable and efficient manner.

Action Code#

import browserless from "../../browserless.app.mjs";

export default {
  key: "browserless-scrape-url-list",
  name: "Scrape URL List",
  description: "Scrape content from a list of pages. [See the documentation](https://www.browserless.io/docs/scrape).",
  version: "0.0.2",
  type: "action",
  props: {
    browserless,
    urls: {
      type: "string[]",
      label: "URLs",
      description: "The list of URLs to scrape content from.",
    },
    selectors: {
      type: "string[]",
      label: "Selectors",
      description: "The HTML tag selectors to extract.",
      default: [
        "body",
      ],
    },
  },
  async run({ $ }) {
    if (typeof this.urls === "string") {
      this.urls = JSON.parse(this.urls);
    }

    const result = {};
    const promises = [];

    for (const url of this.urls) {
      promises.push(this.browserless.scrape({
        $,
        data: {
          url,
          elements: this.selectors?.map((selector) => ({
            selector,
          })),
        },
      }));
    }

    const responses = await Promise.all(promises);
    for (let i = 0; i < promises.length; i++) {
      result[this.urls[i]] = responses[i].data[0].results[0].text;
    }

    return result;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Browserless	`browserless`	`app`	This component uses the Browserless app.
URLs	`urls`	`string[]`	The list of URLs to scrape content from.
Selectors	`selectors`	`string[]`	The HTML tag selectors to extract.

Action Authentication#

Browserless uses API keys for authentication. When you connect your Browserless account, Pipedream securely stores the keys so you can easily authenticate to Browserless APIs in both code and no-code steps.

Sign-up for a browserless account, it will create a unique token that allows service interaction.