Submit Scrape Job with Scrapeless API on New Action Created from Hive API

Pipedream makes it easy to connect APIs for Scrapeless, Hive and 2,500+ other apps remarkably fast.

Trigger workflow on

New Action Created from the Hive API

Next, do this

Submit Scrape Job with the Scrapeless API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Hive trigger and Scrapeless action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Action Created trigger
1. Connect your Hive account
2. Configure Polling interval
3. Optional- Select a Workspace Id
Configure the Submit Scrape Job action
1. Connect your Scrapeless account
2. Select a Actor
3. Optional- Configure Input URL
4. Optional- Configure Proxy Country
5. Optional- Configure Additional Input
6. Configure Async Mode
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Action Created on Hive

Description:Emit new event when a new action is created.

Version:0.0.1

Key:hive-new-action-created

View on GitHub

Hive Overview#

The Hive API allows for robust interaction with Hive's project and team management tools. By leveraging the Hive API within Pipedream, you can automate tasks, sync data across platforms, and create custom workflows that enhance productivity. For instance, you could automate project creation, update task statuses, or sync information to other business tools. Pipedream's serverless platform simplifies integration, enabling you to focus on crafting workflows without worrying about infrastructure.

Trigger Code#

import common from "../common/base.mjs";

export default {
  ...common,
  key: "hive-new-action-created",
  name: "New Action Created",
  version: "0.0.1",
  description: "Emit new event when a new action is created.",
  type: "source",
  dedupe: "unique",
  methods: {
    ...common.methods,
    getFunction(hive) {
      return hive.listActions;
    },
    getSummary(actionId) {
      return `A new action with id: "${actionId}" was created!`;
    },
    getField() {
      return "createdAt";
    },
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Hive	`hive`	`app`	This component uses the Hive app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
Polling interval	`timer`	`$.interface.timer`	Pipedream will poll the Hive on this schedule
Workspace Id	`workspaceId`	`string`	Select a value from the drop down menu.

Trigger Authentication#

Hive uses API keys for authentication. When you connect your Hive account, Pipedream securely stores the keys so you can easily authenticate to Hive APIs in both code and no-code steps.

You can get your API Key, User ID, and Workspace ID in the profile menu on the top right, then Edit Profile > API Info.

About Hive#

An all-in-one solution for software and operations to help you manage your D2C brand throughout fulfillment and beyond.

Action#

Submit Scrape Job on Scrapeless

Description:Submit a new web scraping job with specified target URL and extraction rules. [See the documentation](https://apidocs.scrapeless.com/api-11949852)

Version:0.0.1

Key:scrapeless-submit-scrape-job

View on GitHub

Action Code#

import { ConfigurationError } from "@pipedream/platform";
import { ACTOR_OPTIONS } from "../../common/constants.mjs";
import { parseObject } from "../../common/utils.mjs";
import scrapeless from "../../scrapeless.app.mjs";

export default {
  key: "scrapeless-submit-scrape-job",
  name: "Submit Scrape Job",
  description: "Submit a new web scraping job with specified target URL and extraction rules. [See the documentation](https://apidocs.scrapeless.com/api-11949852)",
  version: "0.0.1",
  type: "action",
  props: {
    scrapeless,
    actor: {
      type: "string",
      label: "Actor",
      description: "The actor to use for the scrape job. This can be a specific user or a system account.",
      options: ACTOR_OPTIONS,
    },
    inputUrl: {
      type: "string",
      label: "Input URL",
      description: "Target URL to scrape. This is the URL of the web page you want to extract data from.",
      optional: true,
    },
    proxyCountry: {
      type: "string",
      label: "Proxy Country",
      description: "The country to route the request through. This can help in bypassing geo-restrictions.",
      optional: true,
    },
    additionalInput: {
      type: "object",
      label: "Additional Input",
      description: "Additional input parameters if you need to pass a specific configuration based on the actor. [See the documentation](https://apidocs.scrapeless.com/) for further details.",
      optional: true,
    },
    asyncMode: {
      type: "boolean",
      label: "Async Mode",
      description: "Whether to run the scrape job in asynchronous mode. If set to true, the job will be processed in the background.",
    },
  },
  async run({ $ }) {
    try {
      const data = {
        actor: this.actor,
        input: parseObject(this.additionalInput),
      };

      if (this.asyncMode) {
        data.async = this.asyncMode;
      }
      if (this.inputUrl) {
        data.input.url = this.inputUrl;
      }
      if (this.proxyCountry) {
        data.proxy = {
          country: this.proxyCountry,
        };
      }

      const response = await this.scrapeless.submitScrapeJob({
        $,
        data,
      });

      $.export("$summary", this.asyncMode
        ? `Successfully submitted scrape job with ID: ${response.taskId}`
        : "Successfully scraped the target configuration.");
      return response;
    } catch ({ response }) {
      throw new ConfigurationError(response.data.message);
    }
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Scrapeless	`scrapeless`	`app`	This component uses the Scrapeless app.
Actor	`actor`	`string`	Select a value from the drop down menu:`{ "label": "Shopee", "value": "scraper.shopee" }{ "label": "BR Sites - Solucoes cnpjreva", "value": "scraper.solucoes" }{ "label": "BR Sites - Solucoes certidaointernet", "value": "scraper.solucoes.certidaointernet" }{ "label": "BR Sites - Servicos receita", "value": "scraper.servicos.receita" }{ "label": "BR Sites - Consopt", "value": "scraper.consopt" }{ "label": "Avnet", "value": "scraper.avnet" }{ "label": "Arrow", "value": "scraper.arrow" }{ "label": "Airline Iberia", "value": "scraper.iberia" }{ "label": "Airline Expedia", "value": "scraper.expedia" }{ "label": "Airline Kayak", "value": "scraper.kayak" }{ "label": "Amazon Product", "value": "scraper.amazon.product" }{ "label": "Amazon Seller", "value": "scraper.amazon.seller" }{ "label": "Amazon Keywords", "value": "scraper.amazon.keywords" }{ "label": "Temu", "value": "scraper.temu.mobile.detail" }{ "label": "Google Search", "value": "scraper.google.search" }{ "label": "Google Trends", "value": "scraper.google.trends" }{ "label": "Google FLights", "value": "scraper.google.flights" }{ "label": "Google FLights Chart", "value": "scraper.google.flights.chart" }{ "label": "Google Maps", "value": "scraper.google.maps" }{ "label": "Google Scholar", "value": "scraper.google.scholar" }{ "label": "Google Jobs", "value": "scraper.google.jobs" }{ "label": "Google Shopping", "value": "scraper.google.shopping" }{ "label": "Google Hotels", "value": "scraper.google.hotels" }{ "label": "Google News", "value": "scraper.google.news" }{ "label": "Google Lens", "value": "scraper.google.lens" }{ "label": "Google Finance", "value": "scraper.google.finance" }{ "label": "Google Product", "value": "scraper.google.product" }{ "label": "Google Play Games", "value": "scraper.google.play.games" }{ "label": "Google Play Books", "value": "scraper.google.play.books" }{ "label": "Google Play Movies", "value": "scraper.google.play.movies" }{ "label": "Google Play Product", "value": "scraper.google.play.product" }{ "label": "Google Play Apps", "value": "scraper.google.play" }{ "label": "Google Ads", "value": "scraper.google.ads" }{ "label": "Mouser", "value": "scraper.mouser" }`
Input URL	`inputUrl`	`string`	Target URL to scrape. This is the URL of the web page you want to extract data from.
Proxy Country	`proxyCountry`	`string`	The country to route the request through. This can help in bypassing geo-restrictions.
Additional Input	`additionalInput`	`object`	Additional input parameters if you need to pass a specific configuration based on the actor. See the documentation for further details.
Async Mode	`asyncMode`	`boolean`	Whether to run the scrape job in asynchronous mode. If set to true, the job will be processed in the background.