Scrape URL List with Browserless API on New Deployment from Vercel API

Pipedream makes it easy to connect APIs for Browserless, Vercel and 2,400+ other apps remarkably fast.

Trigger workflow on

New Deployment from the Vercel API

Next, do this

Scrape URL List with the Browserless API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Vercel trigger and Browserless action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Deployment trigger
1. Connect your Vercel account
2. Configure timer
3. Optional- Select a Project
4. Optional- Select a State
5. Optional- Configure Max
Configure the Scrape URL List action
1. Connect your Browserless account
2. Configure URLs
3. Configure Selectors
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Deployment on Vercel

Description:Emit new event when a deployment is created

Version:0.0.3

Key:vercel_token_auth-new-deployment

View on GitHub

Vercel Overview#

The Vercel API empowers developers to automate, manage, and interact with their Vercel projects and deployments directly through code. With the Vercel API on Pipedream, you can harness the power of serverless functions to create dynamic and responsive workflows. Automate deployment processes, sync deployment statuses with other tools, trigger notifications based on deployment events, or manage your domains and aliases—all within the seamless integration landscape of Pipedream.

Trigger Code#

import vercelTokenAuth from "../../vercel_token_auth.app.mjs";
import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";

export default {
  key: "vercel_token_auth-new-deployment",
  name: "New Deployment",
  description: "Emit new event when a deployment is created",
  version: "0.0.3",
  type: "source",
  dedupe: "unique",
  props: {
    vercelTokenAuth,
    db: "$.service.db",
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    project: {
      propDefinition: [
        vercelTokenAuth,
        "project",
      ],
    },
    state: {
      propDefinition: [
        vercelTokenAuth,
        "state",
      ],
    },
    max: {
      propDefinition: [
        vercelTokenAuth,
        "max",
      ],
    },
  },
  hooks: {
    async deploy() {
      await this.processEvent(10);
    },
  },
  methods: {
    _getFrom() {
      return this.db.get("from");
    },
    _setFrom(from) {
      this.db.set("from", from);
    },
    generateMeta(deployment) {
      const {
        uid,
        name,
        state,
        created,
      } = deployment;
      return {
        id: uid,
        summary: `${name ?? uid} ${state}`,
        ts: created,
      };
    },
    async processEvent(max) {
      const params = {
        projectId: this.project,
        state: this.state,
      };
      let from = this._getFrom();
      if (from) {
        params.from = from;
      }
      const deployments = await this.vercelTokenAuth.listDeployments(params, max);
      for (const deployment of deployments) {
        if (!from || deployment.created > from) {
          from = deployment.created;
        }
        const meta = this.generateMeta(deployment);
        this.$emit(deployment, meta);
      }
      this._setFrom(from);
    },
  },
  async run() {
    await this.processEvent(this.max);
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Vercel	`vercelTokenAuth`	`app`	This component uses the Vercel app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`
Project	`project`	`string`	Select a value from the drop down menu.
State	`state`	`string`	Select a value from the drop down menu:`BUILDINGERRORINITIALIZINGQUEUEDREADYCANCELED`
Max	`max`	`integer`	Maximum number of results to return

Trigger Authentication#

Vercel uses API keys for authentication. When you connect your Vercel account, Pipedream securely stores the keys so you can easily authenticate to Vercel APIs in both code and no-code steps.

Vercel encourages users to connect their account with Pipedream using Personal Access Tokens, which can be managed in your Vercel account here.

To learn more, refer to Vercel's documentation.

About Vercel#

Vercel is the platform for frontend developers, providing the speed and reliability innovators need to create at the moment of inspiration.

Action#

Scrape URL List on Browserless

Description:Scrape content from a list of pages. [See the documentation](https://www.browserless.io/docs/scrape).

Version:0.0.2

Key:browserless-scrape-url-list

View on GitHub

Browserless Overview#

The Browserless API on Pipedream allows you to automate browser actions without the overhead of managing your own browser infrastructure. This service provides a way to run Chrome browser sessions programmatically, making it ideal for web scraping, automated testing, and screenshot capture. Leveraging this on Pipedream, you can create serverless workflows that interact with web pages, extract data, and perform actions as a human would, all in a scalable and efficient manner.

Action Code#

import browserless from "../../browserless.app.mjs";

export default {
  key: "browserless-scrape-url-list",
  name: "Scrape URL List",
  description: "Scrape content from a list of pages. [See the documentation](https://www.browserless.io/docs/scrape).",
  version: "0.0.2",
  type: "action",
  props: {
    browserless,
    urls: {
      type: "string[]",
      label: "URLs",
      description: "The list of URLs to scrape content from.",
    },
    selectors: {
      type: "string[]",
      label: "Selectors",
      description: "The HTML tag selectors to extract.",
      default: [
        "body",
      ],
    },
  },
  async run({ $ }) {
    if (typeof this.urls === "string") {
      this.urls = JSON.parse(this.urls);
    }

    const result = {};
    const promises = [];

    for (const url of this.urls) {
      promises.push(this.browserless.scrape({
        $,
        data: {
          url,
          elements: this.selectors?.map((selector) => ({
            selector,
          })),
        },
      }));
    }

    const responses = await Promise.all(promises);
    for (let i = 0; i < promises.length; i++) {
      result[this.urls[i]] = responses[i].data[0].results[0].text;
    }

    return result;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Browserless	`browserless`	`app`	This component uses the Browserless app.
URLs	`urls`	`string[]`	The list of URLs to scrape content from.
Selectors	`selectors`	`string[]`	The HTML tag selectors to extract.