Start Scraping with Scrape-It.Cloud API on New Tag Added to Forum Thread from Discord Bot API

Pipedream makes it easy to connect APIs for Scrape-It.Cloud, Discord Bot and 2,400+ other apps remarkably fast.

Trigger workflow on

New Tag Added to Forum Thread from the Discord Bot API

Next, do this

Start Scraping with the Scrape-It.Cloud API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Discord Bot trigger and Scrape-It.Cloud action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Tag Added to Forum Thread trigger
1. Connect your Discord Bot account
2. Select a Guild
3. Configure timer
Configure the Start Scraping action
1. Connect your Scrape-It.Cloud account
2. Configure URL
3. Optional- Configure Headers
4. Optional- Configure Block Resources
5. Optional- Configure Block URLs
6. Optional- Configure Wait
7. Optional- Configure Wait For
8. Optional- Configure Screenshot
9. Optional- Configure JS Scenario
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Tag Added to Forum Thread on Discord Bot

Description:Emit new event when a new tag is added to a thread

Version:0.0.1

Key:discord_bot-new-tag-added-to-thread

View on GitHub

Discord Bot Overview#

The Discord Bot API unlocks the power to interact with Discord users and channels programmatically, making it possible to automate messages, manage servers, and integrate with other services. With Pipedream's serverless platform, you can create complex workflows that respond to events in Discord, process data, and trigger actions in other apps. This opens up opportunities for community engagement, content moderation, analytics, and more, without the overhead of managing infrastructure.

Trigger Code#

import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";
import common from "../common.mjs";
import sampleEmit from "./test-event.mjs";

export default {
  ...common,
  key: "discord_bot-new-tag-added-to-thread",
  name: "New Tag Added to Forum Thread",
  description: "Emit new event when a new tag is added to a thread",
  type: "source",
  version: "0.0.1",
  dedupe: "unique",
  props: {
    ...common.props,
    db: "$.service.db",
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
  },
  hooks: {
    async deploy() {
      const tags = {};
      const { threads } = await this.discord.listThreads({
        guildId: this.guildId,
      });
      threads.forEach((thread) => {
        if (thread?.applied_tags) {
          tags[thread.id] = thread?.applied_tags;
        }
      });
      this._setTags(tags);
    },
  },
  methods: {
    ...common.methods,
    _getTags() {
      return this.db.get("tags") || {};
    },
    _setTags(tags) {
      this.db.set("tags", tags);
    },
    generateMeta(thread) {
      return {
        id: thread.id,
        summary: `New tag in thread ${thread.id}`,
        ts: Date.now(),
      };
    },
    getChannel(id) {
      return this.discord._makeRequest({
        path: `/channels/${id}`,
      });
    },
  },
  async run() {
    let tags = this._getTags();

    const { threads } = await this.discord.listThreads({
      guildId: this.guildId,
    });

    for (const thread of threads) {
      if (!thread.applied_tags) {
        continue;
      }
      if (thread.applied_tags.some((tag) => !tags[thread.id] || !tags[thread.id].includes(tag))) {
        tags[thread.id] = thread.applied_tags;

        const { available_tags: availableTags = [] } = await this.getChannel(thread.parent_id);

        thread.applied_tags = thread.applied_tags.map((tagId) => ({
          ...availableTags.find(({ id }) => id === tagId),
        }));

        const meta = this.generateMeta(thread);
        this.$emit(thread, meta);

      }
    }

    this._setTags(tags);
  },
  sampleEmit,
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Discord Bot	`discord`	`app`	This component uses the Discord Bot app.
Guild	`guildId`	`string`	Select a value from the drop down menu.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`

Trigger Authentication#

Discord Bot uses API keys for authentication. When you connect your Discord Bot account, Pipedream securely stores the keys so you can easily authenticate to Discord Bot APIs in both code and no-code steps.

This app allows you to use the Discord API using your own Discord bot. If you don't want to use a custom bot, and you just need to use the Discord API, exit this screen and use the Discord app, instead.

If you want to use your own Discord bot, but haven't created one yet, see these instructions or watch this video. You'll need to add this bot to your server(s) to make successful API requests.

Once you've created your bot, you'll find the Bot token within the Bot section of your app. Enter that token below.

About Discord Bot#

Use this app to interact with the Discord API using a bot in your account

Action#

Start Scraping on Scrape-It.Cloud

Description:Initiate the scraping process for a specific endpoint. [See the documentation here](https://scrape-it.cloud/docs/api-features/basic-request).

Version:0.0.1

Key:scrape_it_cloud-start-scraping

View on GitHub

Scrape-It.Cloud Overview#

The Scrape-It.Cloud API allows you to automate the extraction of data from websites. It can parse, scrape, and retrieve content without the need for manual intervention. With this API on Pipedream, you can build workflows that trigger on various events and use the scraped data for numerous applications like data analysis, lead generation, and content aggregation.

Action Code#

import app from "../../scrape_it_cloud.app.mjs";
import utils from "../../common/utils.mjs";

export default {
  key: "scrape_it_cloud-start-scraping",
  name: "Start Scraping",
  description: "Initiate the scraping process for a specific endpoint. [See the documentation here](https://scrape-it.cloud/docs/api-features/basic-request).",
  type: "action",
  version: "0.0.1",
  props: {
    app,
    url: {
      type: "string",
      label: "URL",
      description: "Specify the URL of the web page you would like to scrape.",
    },
    headers: {
      type: "object",
      label: "Headers",
      description: "Specify custom HTTP headers to be passed to the request. For example, in order to override the User-Agent header, use `User-Agent` as a key and `Teapot` as a value.",
      optional: true,
    },
    blockResources: {
      type: "boolean",
      label: "Block Resources",
      description: "By default Scrape-it.cloud do not block images and CSS in the scraped page. To speed up requests and block images and CSS set this parameter value to true.",
      optional: true,
    },
    blockUrls: {
      type: "string[]",
      label: "Block URLs",
      description: "If you want to block some resources except images and CSS, for example, analytics scripts you can add part of the urls to be blocked. Follow [Blocking URLs](https://scrape-it.cloud/docs/api-features/blocking-urls) page to get more information.",
      optional: true,
    },
    wait: {
      type: "integer",
      label: "Wait",
      description: "Some websites may use javascript frameworks that may require a few extra seconds to load their content. This parameters specifies the time in milliseconds to wait for the website. Recommended values are in the interval `5000` - `10000`.",
      optional: true,
    },
    waitFor: {
      type: "string",
      label: "Wait For",
      description: "Specify a CSS selector and the API will wait 30 seconds until the selector appears.",
      optional: true,
    },
    screenshot: {
      type: "boolean",
      label: "Screenshot",
      description: "If you want to get a screenshot of the page you want to scrape, set screenshot parameter value to `true`.",
      optional: true,
    },
    jsScenario: {
      type: "string[]",
      label: "JS Scenario",
      description: "If you want to do some acts on the pages you want to scrape, for example, go to the next one, set JavaScript scenario to API call. Follow [JavaScript Execution](https://scrape-it.cloud/docs/api-features/javascript-execution) page to get more information.",
      optional: true,
    },
  },
  methods: {
    scrape(args = {}) {
      return this.app.post({
        path: "/scrape",
        ...args,
      });
    },
  },
  async run({ $: step }) {
    const {
      url, headers, blockResources, blockUrls, wait, waitFor, screenshot, jsScenario,
    } = this;

    const response = await this.scrape({
      step,
      data: {
        url,
        headers: utils.parse(headers),
        block_resources: blockResources,
        block_urls: utils.parseArray(blockUrls),
        wait,
        wait_for: waitFor,
        screenshot,
        js_scenario: utils.parseArray(jsScenario)?.map(utils.parse),
      },
    });

    step.export("$summary", `Successfully scraped ${url}.`);

    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Scrape-It.Cloud	`app`	`app`	This component uses the Scrape-It.Cloud app.
URL	`url`	`string`	Specify the URL of the web page you would like to scrape.
Headers	`headers`	`object`	Specify custom HTTP headers to be passed to the request. For example, in order to override the User-Agent header, use `User-Agent` as a key and `Teapot` as a value.
Block Resources	`blockResources`	`boolean`	By default Scrape-it.cloud do not block images and CSS in the scraped page. To speed up requests and block images and CSS set this parameter value to true.
Block URLs	`blockUrls`	`string[]`	If you want to block some resources except images and CSS, for example, analytics scripts you can add part of the urls to be blocked. Follow Blocking URLs page to get more information.
Wait	`wait`	`integer`	Some websites may use javascript frameworks that may require a few extra seconds to load their content. This parameters specifies the time in milliseconds to wait for the website. Recommended values are in the interval `5000` - `10000`.
Wait For	`waitFor`	`string`	Specify a CSS selector and the API will wait 30 seconds until the selector appears.
Screenshot	`screenshot`	`boolean`	If you want to get a screenshot of the page you want to scrape, set screenshot parameter value to `true`.
JS Scenario	`jsScenario`	`string[]`	If you want to do some acts on the pages you want to scrape, for example, go to the next one, set JavaScript scenario to API call. Follow JavaScript Execution page to get more information.

Action Authentication#

Scrape-It.Cloud uses API keys for authentication. When you connect your Scrape-It.Cloud account, Pipedream securely stores the keys so you can easily authenticate to Scrape-It.Cloud APIs in both code and no-code steps.

About Scrape-It.Cloud#

Scrape-It.Cloud is an API for web scraping valuable data in JSON format from any website without needing a proxy.

More Ways to Connect Scrape-It.Cloud + Discord Bot#

Other Popular Integrations#

Start Scraping with Scrape-It.Cloud API on New Guild Member from Discord Bot API

Discord Bot + Scrape-It.Cloud

Try it

Start Scraping with Scrape-It.Cloud API on New Message in Channel from Discord Bot API

Discord Bot + Scrape-It.Cloud

Try it

Start Scraping with Scrape-It.Cloud API on New Thread Message from Discord Bot API

Discord Bot + Scrape-It.Cloud

Try it

Start Scraping with Scrape-It.Cloud API on New Forum Thread Message from Discord Bot API

Discord Bot + Scrape-It.Cloud

Try it

Popular Triggers#

New Message in Channel from the Discord Bot API

Emit new event for each message posted to one or more channels

Try it

New Forum Thread Message from the Discord Bot API

Emit new event for each forum thread message posted. Note that your bot must have the MESSAGE_CONTENT privilege intent to see the message content. See the documentation.

Try it

New Guild Member from the Discord Bot API

Emit new event for every member added to a guild. See docs here

Try it

New Tag Added to Forum Thread from the Discord Bot API

Emit new event when a new tag is added to a thread

Try it

New Thread Message from the Discord Bot API

Emit new event for each thread message posted.

Try it