Scrape Page with Scrapfly API on New Item in Feed from RSS API

Pipedream makes it easy to connect APIs for Scrapfly, RSS and 2,400+ other apps remarkably fast.

Trigger workflow on

New Item in Feed from the RSS API

Next, do this

Scrape Page with the Scrapfly API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a RSS trigger and Scrapfly action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Item in Feed trigger
1. Connect your RSS account
2. Configure timer
3. Configure Feed URL
4. Configure Published After
Configure the Scrape Page action
1. Connect your Scrapfly account
2. Configure URL
3. Optional- Configure Headers
4. Optional- Configure Language
5. Optional- Configure Operating System
6. Optional- Configure Timeout
7. Optional- Select a Format
8. Optional- Configure Retry
9. Optional- Configure Proxified Response
10. Optional- Configure Debug
11. Optional- Configure Correlation ID
12. Optional- Configure Tags
13. Optional- Configure DNS
14. Optional- Configure SSL
15. Optional- Select a Proxy Pool
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Item in Feed on RSS

Description:Emit new items from an RSS feed

Version:1.2.7

Key:rss-new-item-in-feed

View on GitHub

RSS Overview#

The RSS app allows users to automatically fetch and parse updates from web feeds. This functionality is pivotal for staying abreast of content changes or updates from websites, blogs, and news outlets that offer RSS feeds. With Pipedream, you can harness the RSS API to trigger workflows that enable a broad range of automations, like content aggregation, monitoring for specific keywords, notifications, and data synchronization across platforms.

Trigger Code#

import { defineSource } from "@pipedream/types";
import rss from "../../app/rss.app.mjs";
import rssCommon from "../common/common.mjs";
export default defineSource({
    ...rssCommon,
    key: "rss-new-item-in-feed",
    name: "New Item in Feed",
    description: "Emit new items from an RSS feed",
    version: "1.2.7",
    type: "source",
    dedupe: "unique",
    props: {
        ...rssCommon.props,
        url: {
            propDefinition: [
                rss,
                "url",
            ],
        },
        publishedAfter: {
            type: "string",
            label: "Published After",
            description: "Emit items published after the specified date in ISO 8601 format .e.g `2022-12-07T12:57:10+07:00`",
            default: new Date(Date.now() - 24 * 60 * 60 * 1000).toISOString(), // default to one day ago
        },
    },
    hooks: {
        async activate() {
            // Try to parse the feed one time to confirm we can fetch and parse.
            // The code will throw any errors to the user.
            await this.rss.fetchAndParseFeed(this.url);
        },
    },
    methods: {
        ...rssCommon.methods,
        generateMeta: function (item) {
            return {
                id: this.rss.itemKey(item),
                summary: item.title,
                ts: Date.now(),
            };
        },
    },
    async run() {
        const items = await this.rss.fetchAndParseFeed(this.url);
        for (const item of items.reverse()) {
            const publishedAfter = +new Date(this.publishedAfter);
            const ts = this.rss.itemTs(item);
            if (Number.isNaN(publishedAfter) || publishedAfter > ts) {
                continue;
            }
            const meta = this.generateMeta(item);
            this.$emit(item, meta);
        }
    },
});

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
RSS	`rss`	`app`	This component uses the RSS app.
	`timer`	`$.interface.timer`	How often you want to poll the feed for new items
Feed URL	`url`	`string`	Enter the URL for any public RSS feed
Published After	`publishedAfter`	`string`	Emit items published after the specified date in ISO 8601 format .e.g `2022-12-07T12:57:10+07:00`

Trigger Authentication#

The RSS API does not require authentication.

About RSS#

Real Simple Syndication

Action#

Scrape Page on Scrapfly

Description:Extract data from a specified web page. [See the documentation](https://scrapfly.io/docs/scrape-api/getting-started)

Version:0.0.1

Key:scrapfly-scrape-page

View on GitHub

Action Code#

import { ConfigurationError } from "@pipedream/platform";
import {
  FORMAT_OPTIONS,
  PROXY_COUNTRY_OPTIONS,
  PROXY_POOL_OPTIONS,
} from "../../common/constants.mjs";
import { parseObject } from "../../common/utils.mjs";
import scrapfly from "../../scrapfly.app.mjs";

export default {
  key: "scrapfly-scrape-page",
  name: "Scrape Page",
  description: "Extract data from a specified web page. [See the documentation](https://scrapfly.io/docs/scrape-api/getting-started)",
  version: "0.0.1",
  type: "action",
  props: {
    scrapfly,
    url: {
      propDefinition: [
        scrapfly,
        "url",
      ],
    },
    headers: {
      type: "object",
      label: "Headers",
      description: "Pass custom headers to the request.",
      optional: true,
    },
    lang: {
      type: "string",
      label: "Language",
      description: "Select page language. By default it uses the language of the selected proxy location. Behind the scenes, it configures the `Accept-Language` HTTP header. If the website support the language, the content will be in that lang. **Note: you cannot set headers `Accept-Language` header manually**. [See the documentation](https://scrapfly.io/docs/scrape-api/getting-started#spec)",
      optional: true,
    },
    os: {
      type: "string",
      label: "Operating System",
      description: "Operating System, if not selected it's random. **Note: you cannot set os parameter and `User-Agent` header at the same time.** [See the documentation](https://scrapfly.io/docs/scrape-api/getting-started#spec)",
      optional: true,
    },
    timeout: {
      type: "integer",
      label: "Timeout",
      description: "Timeout in milliseconds. It represents the maximum time allowed for Scrapfly to perform the scrape. Since `timeout` is not trivial to understand see our [extended documentation on timeouts](https://scrapfly.io/docs/scrape-api/understand-timeout)",
      optional: true,
    },
    format: {
      type: "string",
      label: "Format",
      description: "Format of the response.",
      options: FORMAT_OPTIONS,
      optional: true,
    },
    retry: {
      type: "boolean",
      label: "Retry",
      description: "Improve reliability with retries on failure.",
      optional: true,
    },
    proxifiedResponse: {
      type: "boolean",
      label: "Proxified Response",
      description: "Return the content of the page directly.",
      optional: true,
    },
    debug: {
      type: "boolean",
      label: "Debug",
      description: "Store the API result and take a screenshot if rendering js is enabled.",
      optional: true,
    },
    correlationId: {
      type: "string",
      label: "Correlation ID",
      description: "Helper ID for correlating a group of scrapes.",
      optional: true,
    },
    tags: {
      type: "string[]",
      label: "Tags",
      description: "Add tags to your scrapes to group them.",
      optional: true,
    },
    dns: {
      type: "boolean",
      label: "DNS",
      description: "Query and retrieve target DNS information.",
      optional: true,
    },
    ssl: {
      type: "boolean",
      label: "SSL",
      description: "SSL option.",
      optional: true,
    },
    proxyPool: {
      type: "string",
      label: "Proxy Pool",
      description: "Select the proxy pool to use.",
      optional: true,
      options: PROXY_POOL_OPTIONS,
      reloadProps: true,
    },
  },
  async additionalProps() {
    const props = {};
    props.country = {
      type: "string",
      label: "Country",
      description: "Proxy country location. If not set it chooses a random location available. A reference to a country must be ISO 3166 alpha-2 (2 letters). The available countries are defined by the proxy pool you use. [See the documentation](https://scrapfly.io/docs/scrape-api/getting-started#spec)",
      optional: true,
      options: PROXY_COUNTRY_OPTIONS[this.proxyPool],
    };
    return props;
  },
  async run({ $ }) {
    try {
      let headers = "";
      if (this.headers) {
        headers = Object.keys(parseObject(this.headers))
          .reduce((acc, key) => {
            acc.push(`headers[${key}]=${encodeURIComponent(this.headers[key])}`);
            return acc;
          }, []);
      }
      const params = {
        url: this.url,
        proxy_pool: this.proxyPool,
        country: this.country,
        lang: this.lang,
        os: this.os,
        timeout: this.timeout,
        format: this.format,
        retry: this.retry,
        proxified_response: this.proxifiedResponse,
        debug: this.debug,
        correlation_id: this.correlationId,
        tags: parseObject(this.tags),
        dns: this.dns,
        ssl: this.ssl,
        headers,
      };

      const response = await this.scrapfly.extractWebPageContent({
        $,
        params,
      });

      $.export("$summary", `Successfully scraped content from ${this.url}`);
      return response;
    } catch ({ response: { data: { message } } }) {
      throw new ConfigurationError(message);
    }
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Scrapfly	`scrapfly`	`app`	This component uses the Scrapfly app.
URL	`url`	`string`	This URL is used to transform any relative URLs in the document into absolute URLs automatically. It can be either the base URL or the exact URL of the document. Must be url encoded.
Headers	`headers`	`object`	Pass custom headers to the request.
Language	`lang`	`string`	Select page language. By default it uses the language of the selected proxy location. Behind the scenes, it configures the `Accept-Language` HTTP header. If the website support the language, the content will be in that lang. Note: you cannot set headers `Accept-Language` header manually. See the documentation
Operating System	`os`	`string`	Operating System, if not selected it's random. Note: you cannot set os parameter and `User-Agent` header at the same time. See the documentation
Timeout	`timeout`	`integer`	Timeout in milliseconds. It represents the maximum time allowed for Scrapfly to perform the scrape. Since `timeout` is not trivial to understand see our extended documentation on timeouts
Format	`format`	`string`	Select a value from the drop down menu:`rawtextmarkdownmarkdown:no_links,no_imagesLLMclean_htmljson`
Retry	`retry`	`boolean`	Improve reliability with retries on failure.
Proxified Response	`proxifiedResponse`	`boolean`	Return the content of the page directly.
Debug	`debug`	`boolean`	Store the API result and take a screenshot if rendering js is enabled.
Correlation ID	`correlationId`	`string`	Helper ID for correlating a group of scrapes.
Tags	`tags`	`string[]`	Add tags to your scrapes to group them.
DNS	`dns`	`boolean`	Query and retrieve target DNS information.
SSL	`ssl`	`boolean`	SSL option.
Proxy Pool	`proxyPool`	`string`	Select a value from the drop down menu:`public_datacenter_poolpublic_residential_pool`