Integrate the Pipedream API with the WebScraper.IO API

Setup the Pipedream API trigger to run a workflow which integrates with the WebScraper.IO API. Pipedream's integration platform allows you to integrate Pipedream and WebScraper.IO remarkably fast. Free for developers.

Popular Ways to Connect Pipedream with WebScraper.IO#

Create Scraping Job with WebScraper.IO API on New Scheduled Tasks from Pipedream API

Pipedream + WebScraper.IO

Try it

Create Sitemap with WebScraper.IO API on New Scheduled Tasks from Pipedream API

Pipedream + WebScraper.IO

Try it

Get Scraping Jobs with WebScraper.IO API on New Scheduled Tasks from Pipedream API

Pipedream + WebScraper.IO

Try it

Create a Subscription with Pipedream API on New Scraping Completed from WebScraper.IO API

WebScraper.IO + Pipedream

Try it

Delete a Subscription with Pipedream API on New Scraping Completed from WebScraper.IO API

WebScraper.IO + Pipedream

Try it

Popular Pipedream and WebScraper.IO Triggers#

New Scheduled Tasks from the Pipedream API

Exposes an HTTP API for scheduling messages to be emitted at a future time

Try it

New Upcoming Event Alert from the Pipedream API

Emit new event based on a time interval before an upcoming event in the calendar. This source uses Pipedream's Task Scheduler. See the documentation for more information and instructions for connecting your Pipedream account.

Try it

New Scraping Completed from the WebScraper.IO API

Emit new event when a page scraping job has completed. See the docs here

Try it

New Upcoming Calendar Event from the Pipedream API

Emit new event when a Calendar event is upcoming, this source is using reminderMinutesBeforeStart property of the event to determine the time it should emit.

Try it

New Upcoming Event Type Alert from the Pipedream API

Emit new event based on a time interval before an upcoming event of the specified type ("default", "focusTime", "outOfOffice", "workingLocation") in the calendar. This source uses Pipedream's Task Scheduler. See the documentation for more information and instructions for connecting your Pipedream account.

Try it

Popular Pipedream and WebScraper.IO Actions#

Create Scraping Job with the WebScraper.IO API

Creates a scraping job (scrapes a sitemap). See the docs here

Try it

Create Sitemap with the WebScraper.IO API

Creates a sitemap for the selected website. See the docs here

Try it

Create a Subscription with the Pipedream API

Create a Subscription. See Doc

Try it

Get Scraping Jobs with the WebScraper.IO API

Retrieves a list of scraping jobs for a sitemap. See the docs here

Try it

Delete a Subscription with the Pipedream API

Delete a Subscription. See Doc

Try it

Overview of Pipedream#

Pipedream is an API that allows you to build applications that can connect to
various data sources and processes them in real-time. You can use Pipedream to
create applications that can perform ETL (Extract, Transform, and Load) tasks,
as well as to create data-driven workflows.

Some examples of applications you can build using the Pipedream API include:

An application that can extract data from a database, transform it, and then
load it into another database.
An application that can monitor a data source for changes, and then trigger a
workflow in response to those changes.
An application that can poll an API for new data, and then process that data
in real-time.

Connect Pipedream#

import { axios } from "@pipedream/platform"
export default defineComponent({
  props: {
    pipedream: {
      type: "app",
      app: "pipedream",
    }
  },
  async run({steps, $}) {
    return await axios($, {
      url: `https://api.pipedream.com/v1/users/me`,
      headers: {
        Authorization: `Bearer ${this.pipedream.$auth.api_key}`,
      },
    })
  },
})

Overview of WebScraper.IO#

The WebScraper.IO API allows you to programmatically perform web scraping tasks, extracting structured data from websites. With the API, you can automate the gathering of web content for analysis, monitoring, and integration with other data sources. In Pipedream, you can leverage this API to build workflows that process, analyze, and act on the data you scrape without writing code for backend infrastructure.

Connect WebScraper.IO#

import { axios } from "@pipedream/platform"
export default defineComponent({
  props: {
    webscraper_io: {
      type: "app",
      app: "webscraper_io",
    }
  },
  async run({steps, $}) {
    return await axios($, {
      url: `https://api.webscraper.io/api/v1/sitemaps`,
      params: {
        api_token: `${this.webscraper_io.$auth.api_key}`,
      },
    })
  },
})