Start Scraping with Scrape-It.Cloud API on New event when the content of the URL changes. from HTTP / Webhook API

Pipedream makes it easy to connect APIs for Scrape-It.Cloud, HTTP / Webhook and 2,400+ other apps remarkably fast.

Trigger workflow on

New event when the content of the URL changes. from the HTTP / Webhook API

Next, do this

Start Scraping with the Scrape-It.Cloud API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a HTTP / Webhook trigger and Scrape-It.Cloud action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New event when the content of the URL changes. trigger
1. Connect your HTTP / Webhook account
2. Configure Watching timer
3. Configure HTTP Request Configuration
4. Configure Emit body only
5. Configure Emit as array
Configure the Start Scraping action
1. Connect your Scrape-It.Cloud account
2. Configure URL
3. Optional- Configure Headers
4. Optional- Configure Block Resources
5. Optional- Configure Block URLs
6. Optional- Configure Wait
7. Optional- Configure Wait For
8. Optional- Configure Screenshot
9. Optional- Configure JS Scenario
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New event when the content of the URL changes. on HTTP / Webhook

Description:Emit new event when the content of the URL changes.

Version:0.0.2

Key:http-watch-url

View on GitHub

HTTP / Webhook Overview#

Build, test, and send HTTP requests without code using your Pipedream workflows. The HTTP / Webhook action is a tool to build HTTP requests with a Postman-like graphical interface.

Point and click HTTP requests

Define the target URL, HTTP verb, headers, query parameters, and payload body without writing custom code.

$A screenshot of Pipedream's HTTP Request Configuration interface with a GET request type selected. The request URL is set to 'https://api.openai.com/v1/models'. The 'Auth' tab is highlighted, indicating that authentication is required for this request. In the headers section, there are two headers configured: 'User-Agent' is set to 'pipedream/1', and 'Authorization' is set to 'Bearer {{openai_api_key}}', showing how the OpenAI account's API key is dynamically inserted into the headers to handle authentication automatically.$

Here's an example workflow that uses the HTTP / Webhook action to send an authenticated API request to OpenAI.

Focus on integrating, not authenticating

This action can also use your connected accounts with third-party APIs. Selecting an integrated app will automatically update the request’s headers to authenticate with the app properly, and even inject your token dynamically.

Pipedream integrates with thousands of APIs, but if you can’t find a Pipedream integration simply use Environment Variables in your request headers to authenticate with.

Compatible with no code actions or Node.js and Python

The HTTP/Webhook action exports HTTP response data for use in subsequent workflow steps, enabling easy data transformation, further API calls, database storage, and more.

Response data is available for both coded (Node.js, Python) and no-code steps within your workflow.

Trigger Code#

import http from "../../http.app.mjs";
import hash from "object-hash";
import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";

export default {
  key: "http-watch-url",
  name: "New event when the content of the URL changes.",
  description: "Emit new event when the content of the URL changes.",
  version: "0.0.2",
  type: "source",
  dedupe: "unique",
  props: {
    http,
    timer: {
      type: "$.interface.timer",
      label: "Watching timer",
      description: "How often to watch the URL.",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    httpRequest: {
      type: "http_request",
      label: "HTTP Request Configuration",
      description: "HTTP Request Configuration",
      default: {
        method: "GET",
      },
    },
    emitBodyOnly: {
      label: "Emit body only",
      description: "If set as true the emitted item will contain only the response body, otherwise, it will include the request status code.",
      type: "boolean",
      default: true,
    },
    emitAsArray: {
      label: "Emit as array",
      description: "If the request responds with an array, this source will emit changes individually for each item of the array.",
      type: "boolean",
      default: false,
    },
  },
  methods: {
    getMeta(data, status) {
      if (this.emitBodyOnly) {
        return data;
      }
      return {
        data,
        status,
      };
    },
    emitArray(event, response) {
      if (!Array.isArray(response.data)) {
        return this.emitAny(event, response);
      }
      for (const item of response.data) {
        const meta = this.getMeta(item, response.status);
        this.emit(event, meta);
      }
    },
    emitAny(event, response) {
      const meta = this.getMeta(response.data, response.status);
      this.emit(event, meta);
    },
    emit(event, meta) {
      const ts = event.timestamp || Math.round(Date.now() / 1000);
      this.$emit(meta, {
        id: hash(meta),
        summary: `Requested at ${ts}`,
        ts,
      });
    },
  },
  async run(event) {
    const response = await this.httpRequest.execute();
    if (this.emitAsArray) {
      this.emitArray(event, response);
    } else {
      this.emitAny(event, response);
    }
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
HTTP / Webhook	`http`	`app`	This component uses the HTTP / Webhook app.
Watching timer	`timer`	`$.interface.timer`	How often to watch the URL.
HTTP Request Configuration	`httpRequest`	`http_request`	HTTP Request Configuration
Emit body only	`emitBodyOnly`	`boolean`	If set as true the emitted item will contain only the response body, otherwise, it will include the request status code.
Emit as array	`emitAsArray`	`boolean`	If the request responds with an array, this source will emit changes individually for each item of the array.

Trigger Authentication#

The HTTP / Webhook API does not require authentication.

About HTTP / Webhook#

Get a unique URL where you can send HTTP or webhook requests

Action#

Start Scraping on Scrape-It.Cloud

Description:Initiate the scraping process for a specific endpoint. [See the documentation here](https://scrape-it.cloud/docs/api-features/basic-request).

Version:0.0.1

Key:scrape_it_cloud-start-scraping

View on GitHub

Scrape-It.Cloud Overview#

The Scrape-It.Cloud API allows you to automate the extraction of data from websites. It can parse, scrape, and retrieve content without the need for manual intervention. With this API on Pipedream, you can build workflows that trigger on various events and use the scraped data for numerous applications like data analysis, lead generation, and content aggregation.

Action Code#

import app from "../../scrape_it_cloud.app.mjs";
import utils from "../../common/utils.mjs";

export default {
  key: "scrape_it_cloud-start-scraping",
  name: "Start Scraping",
  description: "Initiate the scraping process for a specific endpoint. [See the documentation here](https://scrape-it.cloud/docs/api-features/basic-request).",
  type: "action",
  version: "0.0.1",
  props: {
    app,
    url: {
      type: "string",
      label: "URL",
      description: "Specify the URL of the web page you would like to scrape.",
    },
    headers: {
      type: "object",
      label: "Headers",
      description: "Specify custom HTTP headers to be passed to the request. For example, in order to override the User-Agent header, use `User-Agent` as a key and `Teapot` as a value.",
      optional: true,
    },
    blockResources: {
      type: "boolean",
      label: "Block Resources",
      description: "By default Scrape-it.cloud do not block images and CSS in the scraped page. To speed up requests and block images and CSS set this parameter value to true.",
      optional: true,
    },
    blockUrls: {
      type: "string[]",
      label: "Block URLs",
      description: "If you want to block some resources except images and CSS, for example, analytics scripts you can add part of the urls to be blocked. Follow [Blocking URLs](https://scrape-it.cloud/docs/api-features/blocking-urls) page to get more information.",
      optional: true,
    },
    wait: {
      type: "integer",
      label: "Wait",
      description: "Some websites may use javascript frameworks that may require a few extra seconds to load their content. This parameters specifies the time in milliseconds to wait for the website. Recommended values are in the interval `5000` - `10000`.",
      optional: true,
    },
    waitFor: {
      type: "string",
      label: "Wait For",
      description: "Specify a CSS selector and the API will wait 30 seconds until the selector appears.",
      optional: true,
    },
    screenshot: {
      type: "boolean",
      label: "Screenshot",
      description: "If you want to get a screenshot of the page you want to scrape, set screenshot parameter value to `true`.",
      optional: true,
    },
    jsScenario: {
      type: "string[]",
      label: "JS Scenario",
      description: "If you want to do some acts on the pages you want to scrape, for example, go to the next one, set JavaScript scenario to API call. Follow [JavaScript Execution](https://scrape-it.cloud/docs/api-features/javascript-execution) page to get more information.",
      optional: true,
    },
  },
  methods: {
    scrape(args = {}) {
      return this.app.post({
        path: "/scrape",
        ...args,
      });
    },
  },
  async run({ $: step }) {
    const {
      url, headers, blockResources, blockUrls, wait, waitFor, screenshot, jsScenario,
    } = this;

    const response = await this.scrape({
      step,
      data: {
        url,
        headers: utils.parse(headers),
        block_resources: blockResources,
        block_urls: utils.parseArray(blockUrls),
        wait,
        wait_for: waitFor,
        screenshot,
        js_scenario: utils.parseArray(jsScenario)?.map(utils.parse),
      },
    });

    step.export("$summary", `Successfully scraped ${url}.`);

    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Scrape-It.Cloud	`app`	`app`	This component uses the Scrape-It.Cloud app.
URL	`url`	`string`	Specify the URL of the web page you would like to scrape.
Headers	`headers`	`object`	Specify custom HTTP headers to be passed to the request. For example, in order to override the User-Agent header, use `User-Agent` as a key and `Teapot` as a value.
Block Resources	`blockResources`	`boolean`	By default Scrape-it.cloud do not block images and CSS in the scraped page. To speed up requests and block images and CSS set this parameter value to true.
Block URLs	`blockUrls`	`string[]`	If you want to block some resources except images and CSS, for example, analytics scripts you can add part of the urls to be blocked. Follow Blocking URLs page to get more information.
Wait	`wait`	`integer`	Some websites may use javascript frameworks that may require a few extra seconds to load their content. This parameters specifies the time in milliseconds to wait for the website. Recommended values are in the interval `5000` - `10000`.
Wait For	`waitFor`	`string`	Specify a CSS selector and the API will wait 30 seconds until the selector appears.
Screenshot	`screenshot`	`boolean`	If you want to get a screenshot of the page you want to scrape, set screenshot parameter value to `true`.
JS Scenario	`jsScenario`	`string[]`	If you want to do some acts on the pages you want to scrape, for example, go to the next one, set JavaScript scenario to API call. Follow JavaScript Execution page to get more information.

Action Authentication#

Scrape-It.Cloud uses API keys for authentication. When you connect your Scrape-It.Cloud account, Pipedream securely stores the keys so you can easily authenticate to Scrape-It.Cloud APIs in both code and no-code steps.

About Scrape-It.Cloud#

Scrape-It.Cloud is an API for web scraping valuable data in JSON format from any website without needing a proxy.

More Ways to Connect Scrape-It.Cloud + HTTP / Webhook#

Other Popular Integrations#

Start Scraping with Scrape-It.Cloud API on New Requests from HTTP / Webhook API

HTTP / Webhook + Scrape-It.Cloud

Try it

Start Scraping with Scrape-It.Cloud API on New Requests (Payload Only) from HTTP / Webhook API

HTTP / Webhook + Scrape-It.Cloud

Try it

Popular Triggers#

New Requests from the HTTP / Webhook API

Get a URL and emit the full HTTP event on every request (including headers and query parameters). You can also configure the HTTP response code, body, and more.

Try it

New Requests (Payload Only) from the HTTP / Webhook API

Get a URL and emit the HTTP body as an event on every request

Try it

New event when the content of the URL changes. from the HTTP / Webhook API

Emit new event when the content of the URL changes.

Try it