Get Scraping Jobs with WebScraper.IO API

Pipedream makes it easy to connect APIs for WebScraper.IO and 2,200+ other apps remarkably fast.

Trigger workflow on

HTTP requests, schedules and app events

Next, do this

Get Scraping Jobs with the WebScraper.IO API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

Create a workflow to Get Scraping Jobs with the WebScraper.IO API. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Configure the Get Scraping Jobs action
1. Connect your WebScraper.IO account
2. Select a Sitemap ID
3. Optional- Configure Max Results
Select a trigger to run your workflow on HTTP requests, schedules or app events
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Integrations#

Get Scraping Jobs with WebScraper.IO API on New Requests from HTTP / Webhook API

HTTP / Webhook + WebScraper.IO

Try it

Get Scraping Jobs with WebScraper.IO API on New Submission from Typeform API

Typeform + WebScraper.IO

Try it

Get Scraping Jobs with WebScraper.IO API on New Submission (Instant) from Jotform API

Jotform + WebScraper.IO

Try it

Get Scraping Jobs with WebScraper.IO API on New Custom Webhook Events from Stripe API

Stripe + WebScraper.IO

Try it

Get Scraping Jobs with WebScraper.IO API on New Scheduled Tasks from Pipedream API

Pipedream + WebScraper.IO

Try it

Details#

This is a pre-built, source-available component from Pipedream's GitHub repo. The component is developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Get Scraping Jobs on WebScraper.IO

Description:Retrieves a list of scraping jobs for a sitemap. [See the docs here](https://webscraper.io/documentation/web-scraper-cloud/api)

Version:0.0.1

Key:webscraper_io-get-scraping-jobs

View on GitHub

Code#

import webscraper from "../../webscraper_io.app.mjs";

export default {
  key: "webscraper_io-get-scraping-jobs",
  name: "Get Scraping Jobs",
  description: "Retrieves a list of scraping jobs for a sitemap. [See the docs here](https://webscraper.io/documentation/web-scraper-cloud/api)",
  version: "0.0.1",
  type: "action",
  props: {
    webscraper,
    sitemapId: {
      propDefinition: [
        webscraper,
        "sitemapId",
      ],
    },
    maxResults: {
      type: "integer",
      label: "Max Results",
      description: "The maximum number of jobs to return",
      default: 100,
      optional: true,
    },
  },
  async run({ $ }) {
    const jobs = await this.webscraper.paginate(this.webscraper.getScrapingJobs, {
      sitemap_id: this.sitemapId,
    }, this.maxResults);

    $.export("$summary", `Successfully retrieved ${jobs.length} scraping job${jobs.length === 1
      ? ""
      : "s"}`);

    return jobs;
  },
};

Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
WebScraper.IO	`webscraper`	`app`	This component uses the WebScraper.IO app.
Sitemap ID	`sitemapId`	`string`	Select a value from the drop down menu.
Max Results	`maxResults`	`integer`	The maximum number of jobs to return

Authentication#

WebScraper.IO uses API keys for authentication. When you connect your WebScraper.IO account, Pipedream securely stores the keys so you can easily authenticate to WebScraper.IO APIs in both code and no-code steps.

" To retrieve your API token,