Request Social Media Scraping with ScrapingBot API on New Direct Message (Instant) from Slack API

Pipedream makes it easy to connect APIs for ScrapingBot, Slack and 2,400+ other apps remarkably fast.

Trigger workflow on

New Direct Message (Instant) from the Slack API

Next, do this

Request Social Media Scraping with the ScrapingBot API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Slack trigger and ScrapingBot action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Direct Message (Instant) trigger
1. Connect your Slack account
2. Configure slackApphook
3. Optional- Configure Ignore Bots
4. Optional- Configure Ignore Messages from Yourself
Configure the Request Social Media Scraping action
1. Connect your ScrapingBot account
2. Select a Scraper
3. Optional- Configure URL
4. Optional- Configure Account
5. Optional- Configure Search
6. Optional- Configure Hashtag
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Direct Message (Instant) on Slack

Description:Emit new event when a message was posted in a direct message channel

Version:1.0.20

Key:slack-new-direct-message

View on GitHub

Slack Overview#

The Pipedream Slack app enables you to build event-driven workflows that interact with the Slack API. Once you authorize the Pipedream app's access to your workspace, you can use Pipedream workflows to perform common Slack actions or write your own code against the Slack API.

The Pipedream Slack app is not a typical app. You don't interact with it directly as a bot, and it doesn't add custom functionality to your workspace out of the box. It makes it easier to automate anything you'd typically use the Slack API for, using Pipedream workflows.

Automate posting updates to your team channels
Create a bot to answer common questions
Integrate with your existing tools and services
And much more!

Trigger Code#

import common from "../common/base.mjs";
import sampleEmit from "./test-event.mjs";

export default {
  ...common,
  key: "slack-new-direct-message",
  name: "New Direct Message (Instant)",
  version: "1.0.20",
  description: "Emit new event when a message was posted in a direct message channel",
  type: "source",
  dedupe: "unique",
  props: {
    ...common.props,
    // eslint-disable-next-line pipedream/props-description,pipedream/props-label
    slackApphook: {
      type: "$.interface.apphook",
      appProp: "slack",
      async eventNames() {
        return [
          "message.im",
        ];
      },
    },
    ignoreBot: {
      propDefinition: [
        common.props.slack,
        "ignoreBot",
      ],
    },
    ignoreSelf: {
      type: "boolean",
      label: "Ignore Messages from Yourself",
      description: "Ignores messages sent to yourself",
      default: false,
      optional: true,
    },
  },
  methods: {
    ...common.methods,
    getSummary() {
      return "New direct message received";
    },
    processEvent(event) {
      if ((this.ignoreSelf && event.user == this.slack.mySlackId())
        || ((this.ignoreBot) && (event.subtype === "bot_message" || event.bot_id))
        || (event.subtype === "message_changed")) {
        return;
      }
      return event;
    },
  },
  sampleEmit,
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Slack	`slack`	`app`	This component uses the Slack app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`slackApphook`	`$.interface.apphook`
Ignore Bots	`ignoreBot`	`boolean`	Ignore messages from bots
Ignore Messages from Yourself	`ignoreSelf`	`boolean`	Ignores messages sent to yourself

Trigger Authentication#

Slack uses OAuth authentication. When you connect your Slack account, Pipedream will open a popup window where you can sign into Slack and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Slack API.

Pipedream requests the following authorization scopes when you connect your account:

channels:historychannels:readchannels:writeemoji:readgroups:historygroups:readgroups:writeim:historyim:readim:writelinks:readlinks:writempim:historympim:readmpim:writereactions:readreactions:writereminders:readreminders:writestars:readteam:readusergroups:readusergroups:writeusers:readusers:read.emailusers:writechat:write:botchat:write:userfiles:write:usercommandsusers.profile:writeusers.profile:read

About Slack#

Slack is a channel-based messaging platform. With Slack, people can work together more effectively, connect all their software tools and services, and find the information they need to do their best work — all within a secure, enterprise-grade environment.

Action#

Request Social Media Scraping on ScrapingBot

Description:Use ScrapingBot API to initiate scraping data from a social media site. [See the documentation](https://www.scraping-bot.io/web-scraping-documentation/)

Version:0.0.1

Key:scrapingbot-request-social-media-scraping

View on GitHub

ScrapingBot Overview#

ScrapingBot API on Pipedream allows you to scrape websites without getting blocked, fetching crucial information while bypassing common defenses. Whether you're extracting product details, real estate listings, or automating competitor research, this API combined with Pipedream's serverless platform offers you the tools to automate these tasks efficiently. Pipedream's ability to trigger workflows via HTTP requests, schedule them, or react to events, means you can create robust scraping operations that integrate seamlessly with hundreds of other apps.

Action Code#

import scrapingbot from "../../scrapingbot.app.mjs";
import { ConfigurationError } from "@pipedream/platform";
import constants from "../../common/constants.mjs";

export default {
  key: "scrapingbot-request-social-media-scraping",
  name: "Request Social Media Scraping",
  description: "Use ScrapingBot API to initiate scraping data from a social media site. [See the documentation](https://www.scraping-bot.io/web-scraping-documentation/)",
  version: "0.0.1",
  type: "action",
  props: {
    scrapingbot,
    scraper: {
      propDefinition: [
        scrapingbot,
        "scraper",
      ],
    },
    url: {
      type: "string",
      label: "URL",
      description: "URL of the page you want to scrape",
      optional: true,
    },
    account: {
      type: "string",
      label: "Account",
      description: "The account name you want to extract data from",
      optional: true,
    },
    search: {
      propDefinition: [
        scrapingbot,
        "search",
      ],
      optional: true,
    },
    hashtag: {
      type: "string",
      label: "Hashtag",
      description: "The hashtag you wish to extract data from",
      optional: true,
    },
  },
  async run({ $ }) {
    const {
      scraper,
      url,
      account,
      search,
      hashtag,
    } = this;

    if (!url && constants.REQUIRES_URL.includes(scraper)) {
      throw new ConfigurationError(`URL is required for scraper ${scraper}`);
    }

    if (!account && constants.REQUIRES_ACCOUNT.includes(scraper)) {
      throw new ConfigurationError(`Account is required for scraper ${scraper}`);
    }

    if (!search && constants.REQUIRES_SEARCH.includes(scraper)) {
      throw new ConfigurationError(`Search is required for scraper ${scraper}`);
    }

    if (!hashtag && constants.REQUIRES_HASHTAG.includes(scraper)) {
      throw new ConfigurationError(`Hashtag is required for scraper ${scraper}`);
    }

    const response = await this.scrapingbot.scrapeSocialMedia({
      data: {
        scraper,
        url,
        account,
        search,
        hashtag,
      },
      $,
    });

    if (response.responseId) {
      $.export("$summary", `Successfully initiated scraping with responseId ${response.responseId}. Scraping Job may take a few minutes to complete.`);
    }
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
ScrapingBot	`scrapingbot`	`app`	This component uses the ScrapingBot app.
Scraper	`scraper`	`string`	Select a value from the drop down menu:`linkedinProfilelinkedinCompanyProfilelinkedinPostinstagramProfileinstagramPostinstagramHashtagfacebookProfilefacebookPostfacebookOrganizationtwitterProfiletwitterSearchtiktokProfiletiktokHashtag`
URL	`url`	`string`	URL of the page you want to scrape
Account	`account`	`string`	The account name you want to extract data from
Search	`search`	`string`	The term to search for
Hashtag	`hashtag`	`string`	The hashtag you wish to extract data from

Request Social Media Scraping with ScrapingBot API on New Direct Message (Instant) from Slack API

Pipedream makes it easy to connect APIs for ScrapingBot, Slack and 2,400+ other apps remarkably fast.

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

1-24of2,400+apps by most popular

1
-
24
of
2,400+
apps by most popular