Scrape without JS with ScrapeNinja API on New Bookmark from Raindrop API

Pipedream makes it easy to connect APIs for ScrapeNinja, Raindrop and 2,900+ other apps remarkably fast.

Trigger workflow on

New Bookmark from the Raindrop API

Next, do this

Scrape without JS with the ScrapeNinja API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Raindrop trigger and ScrapeNinja action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Bookmark trigger
1. Connect your Raindrop account
2. Configure timer
3. Select a Collection ID
Configure the Scrape without JS action
1. Connect your ScrapeNinja account
2. Configure URL
3. Optional- Configure Headers
4. Optional- Configure Retry Number
5. Optional- Configure Geo
6. Optional- Configure Proxy
7. Optional- Configure Follow Redirects
8. Optional- Configure Timeout
9. Optional- Configure Text Not Expected
10. Optional- Configure Status Not Expected
11. Optional- Configure Extractor
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Bookmark on Raindrop

Description:Emit new event when a bookmark is added

Version:0.0.6

Key:raindrop-new-bookmark

View on GitHub

Raindrop Overview#

Raindrop.io is a versatile bookmarking service that lets users save, organize, and share web content. With its API, you can automate the management of bookmarks, collections, and tags, thus streamlining the way you handle internet resources. Automating these tasks with the Raindrop API on Pipedream opens the door to personalized content curation, enhanced collaboration, and efficient information retrieval workflows. Think syncing saved articles across platforms, generating reports on saved links, or even triggering actions based on new content.

Trigger Code#

import raindrop from "../../raindrop.app.mjs";
import constants from "../../common/constants.mjs";
import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";

export default {
  key: "raindrop-new-bookmark",
  name: "New Bookmark",
  description: "Emit new event when a bookmark is added",
  type: "source",
  version: "0.0.6",
  dedupe: "unique",
  props: {
    raindrop,
    db: "$.service.db",
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    collectionId: {
      propDefinition: [
        raindrop,
        "collectionId",
      ],
    },
  },
  methods: {
    _getPage() {
      return this.db.get("page") ?? 0;
    },
    _setPage(page) {
      this.db.set("page", page);
    },
    getMetadata(bookmark) {
      return {
        id: bookmark._id,
        summary: `New Raindrop: ${bookmark.title}`,
        ts: Date.parse(bookmark.created),
      };
    },
    emitEvents(bookmarks) {
      bookmarks.forEach((bookmark) => {
        const meta = this.getMetadata(bookmark);
        this.$emit(bookmark, meta);
      });
    },
  },
  async run() {
    let page = this._getPage();
    const createdDate = `created:>${new Date().toISOString()
      .slice(0, 7)}`;

    while (true) {
      const { items: bookmarks } = await this.raindrop.getRaindrops(this, this.collectionId, {
        page,
        perpage: constants.DEFAULT_PER_PAGE,
        search: createdDate,
      });
      this.emitEvents(bookmarks);

      if (bookmarks.length < constants.DEFAULT_PER_PAGE) break;

      page++;
    }

    this._setPage(page);
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Raindrop	`raindrop`	`app`	This component uses the Raindrop app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`
Collection ID	`collectionId`	`string`	Select a value from the drop down menu.

Trigger Authentication#

Raindrop uses OAuth authentication. When you connect your Raindrop account, Pipedream will open a popup window where you can sign into Raindrop and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Raindrop API.

Pipedream requests the following authorization scopes when you connect your account:

About Raindrop#

Raindrop.io is the best place to keep all your favorite books, songs, articles or whatever else you come across while browsing.

Action#

Scrape without JS on ScrapeNinja

Description:Use high-performance web scraping endpoint with Chrome browser TLS fingerprint, but without JavaScript execution and real browser overhead. [See the documentation](https://scrapeninja.net/docs/api-reference/scrape/)

Version:0.0.2

Key:scrapeninja-scrape-without-js

View on GitHub

ScrapeNinja Overview#

ScrapeNinja API on Pipedream allows you to craft powerful serverless workflows for web scraping without the hassle of managing proxies or browsers. It's a tool that can extract data from websites, handling JavaScript rendering and anti-bot measures with ease. By integrating ScrapeNinja with Pipedream, you can automate data collection, collate and process the scraped data, and connect it to numerous other services for further analysis, alerting, or storage.

Action Code#

import { ConfigurationError } from "@pipedream/platform";
import { parseObject } from "../../common/utils.mjs";
import scrapeninja from "../../scrapeninja.app.mjs";

export default {
  key: "scrapeninja-scrape-without-js",
  name: "Scrape without JS",
  description: "Use high-performance web scraping endpoint with Chrome browser TLS fingerprint, but without JavaScript execution and real browser overhead. [See the documentation](https://scrapeninja.net/docs/api-reference/scrape/)",
  version: "0.0.2",
  annotations: {
    destructiveHint: false,
    openWorldHint: true,
    readOnlyHint: false,
  },
  type: "action",
  props: {
    scrapeninja,
    url: {
      propDefinition: [
        scrapeninja,
        "url",
      ],
    },
    headers: {
      propDefinition: [
        scrapeninja,
        "headers",
      ],
      optional: true,
    },
    retryNum: {
      propDefinition: [
        scrapeninja,
        "retryNum",
      ],
      optional: true,
    },
    geo: {
      propDefinition: [
        scrapeninja,
        "geo",
      ],
      optional: true,
    },
    proxy: {
      propDefinition: [
        scrapeninja,
        "proxy",
      ],
      optional: true,
    },
    followRedirects: {
      propDefinition: [
        scrapeninja,
        "followRedirects",
      ],
      optional: true,
    },
    timeout: {
      propDefinition: [
        scrapeninja,
        "timeout",
      ],
      optional: true,
    },
    textNotExpected: {
      propDefinition: [
        scrapeninja,
        "textNotExpected",
      ],
      optional: true,
    },
    statusNotExpected: {
      propDefinition: [
        scrapeninja,
        "statusNotExpected",
      ],
      optional: true,
    },
    extractor: {
      propDefinition: [
        scrapeninja,
        "extractor",
      ],
      optional: true,
    },
  },
  async run({ $ }) {
    try {
      const response = await this.scrapeninja.scrapeNonJs({
        $,
        data: {
          url: this.url,
          headers: parseObject(this.headers),
          retryNum: this.retryNum,
          geo: this.geo,
          proxy: this.proxy,
          followRedirects: this.followRedirects,
          timeout: this.timeout,
          textNotExpected: parseObject(this.textNotExpected),
          statusNotExpected: parseObject(this.statusNotExpected),
          extractor: this.extractor,
        },
      });
      $.export("$summary", "Successfully scraped the URL");
      return response;
    } catch ({ response: { data } }) {
      throw new ConfigurationError(data.message || data.stderr);
    }
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
ScrapeNinja	`scrapeninja`	`app`	This component uses the ScrapeNinja app.
URL	`url`	`string`	The URL to scrape.
Headers	`headers`	`string[]`	Custom headers to send with the request. By default, regular Chrome browser headers are sent to the target URL.
Retry Number	`retryNum`	`integer`	Amount of attempts.
Geo	`geo`	`string`	Geo location for basic proxy pools (you can purchase premium ScrapeNinja proxies for wider country selection and higher proxy quality). Read more about ScrapeNinja proxy setup
Proxy	`proxy`	`string`	Premium or your own proxy URL (overrides `Geo` prop). Read more about ScrapeNinja proxy setup
Follow Redirects	`followRedirects`	`integer`	Whether to follow redirects.
Timeout	`timeout`	`integer`	Timeout per attempt, in seconds. Each retry will take [timeout] number of seconds.
Text Not Expected	`textNotExpected`	`string[]`	Text which will trigger a retry from another proxy address.
Status Not Expected	`statusNotExpected`	`integer[]`	HTTP response statuses which will trigger a retry from another proxy address.
Extractor	`extractor`	`string`	Custom JS function to extract JSON values from scraped HTML. Write&test your own extractor on https://scrapeninja.net/cheerio-sandbox/

Action Authentication#

ScrapeNinja uses API keys for authentication. When you connect your ScrapeNinja account, Pipedream securely stores the keys so you can easily authenticate to ScrapeNinja APIs in both code and no-code steps.

Using ScrapingNinja in Pipedream

Create a RapidAPI Account: Begin by signing up for a RapidAPI account.
Access Your API Key:
- Once registered, you'll be able to interact with ScrapingNinja using your RapidAPI key.
- Open the ScrapingNinja documentation on RapidAPI and locate your API key labeled X-RapidAPI-Key.
- Copy this key and paste it into the rapid_api_key field below.
Subscribe to the API: Finally, click Subscribe to Test in the RapidAPI console to subscribe to the ScrapingNinja API.