Crawl URL with FireCrawl API on New Team from Microsoft Teams API

Pipedream makes it easy to connect APIs for FireCrawl, Microsoft Teams and 2,800+ other apps remarkably fast.

Trigger workflow on

New Team from the Microsoft Teams API

Next, do this

Crawl URL with the FireCrawl API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Microsoft Teams trigger and FireCrawl action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Team trigger
1. Connect your Microsoft Teams account
2. Configure timer
Configure the Crawl URL action
1. Connect your FireCrawl account
2. Configure URL
3. Optional- Configure Prompt
4. Optional- Configure Exclude Paths
5. Optional- Configure Include Paths
6. Optional- Configure Max Discovery Depth
7. Optional- Select a Sitemap
8. Optional- Configure Ignore Query Parameters
9. Optional- Configure Limit
10. Optional- Configure Crawl Entire Domain
11. Optional- Configure Allow External Links
12. Optional- Configure Additional Options
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Team on Microsoft Teams

Description:Emit new event when a new team is joined by the authenticated user

Version:0.0.10

Key:microsoft_teams-new-team

View on GitHub

Microsoft Teams Overview#

The Microsoft Teams API on Pipedream allows you to automate tasks, streamline communication, and integrate with other services to enhance the functionality of Teams as a collaboration hub. With this API, you can send messages to channels, orchestrate complex workflows based on Teams events, and manage Teams' settings programmatically.

Trigger Code#

import base from "../common/base.mjs";

export default {
  ...base,
  key: "microsoft_teams-new-team",
  name: "New Team",
  description: "Emit new event when a new team is joined by the authenticated user",
  version: "0.0.10",
  type: "source",
  dedupe: "unique",
  methods: {
    ...base.methods,
    async getResources(lastCreated) {
      return this.getNewPaginatedResources(
        this.microsoftTeams.listTeams,
        {},
        lastCreated,
      );
    },
    generateMeta(team) {
      return {
        id: team.id,
        summary: team.displayName,
        ts: Date.now(),
      };
    },
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Microsoft Teams	`microsoftTeams`	`app`	This component uses the Microsoft Teams app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`

Trigger Authentication#

Microsoft Teams uses OAuth authentication. When you connect your Microsoft Teams account, Pipedream will open a popup window where you can sign into Microsoft Teams and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Microsoft Teams API.

Pipedream requests the following authorization scopes when you connect your account:

User.Reademailoffline_accessopenidprofileChat.ReadChat.ReadWriteChatMessage.SendChannel.ReadBasic.AllChannelMessage.Read.AllChannelMessage.SendTeam.ReadBasic.AllSchedule.Read.AllOnlineMeetings.ReadWrite

About Microsoft Teams#

Microsoft Teams has communities, events, chats, channels, meetings, storage, tasks, and calendars in one place.

Action#

Crawl URL on FireCrawl

Description:Crawls a given URL and returns the contents of sub-pages. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post)

Version:1.1.0

Key:firecrawl-crawl-url

View on GitHub

Action Code#

import { parseObjectEntries } from "../../common/utils.mjs";
import firecrawl from "../../firecrawl.app.mjs";

export default {
  key: "firecrawl-crawl-url",
  name: "Crawl URL",
  description: "Crawls a given URL and returns the contents of sub-pages. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post)",
  version: "1.1.0",
  type: "action",
  props: {
    firecrawl,
    url: {
      propDefinition: [
        firecrawl,
        "url",
      ],
    },
    prompt: {
      type: "string",
      label: "Prompt",
      description: "A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents.",
      optional: true,
    },
    excludePaths: {
      type: "string[]",
      label: "Exclude Paths",
      description: "URL pathname regex patterns that exclude matching URLs from the crawl. For example, a value of `blog/.*` for the URL `firecrawl.dev` will exclude any results matching that pattern, such as `https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap`",
      optional: true,
    },
    includePaths: {
      type: "string[]",
      label: "Include Paths",
      description: "Similar to `Exclude Paths`, but if set, only the paths matching the specified patterns will be included",
      optional: true,
    },
    maxDiscoveryDepth: {
      type: "integer",
      label: "Max Discovery Depth",
      description: "Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set sitemap: 'skip', you will only crawl the entered URL and all URLs that are linked on that page.",
      optional: true,
    },
    sitemap: {
      type: "string",
      label: "Sitemap",
      description: "Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards.",
      options: [
        "skip",
        "include",
      ],
      optional: true,
    },
    ignoreQueryParameters: {
      type: "boolean",
      label: "Ignore Query Parameters",
      description: "Do not re-scrape the same path with different (or none) query parameters",
      optional: true,
    },
    limit: {
      type: "integer",
      label: "Limit",
      description: "Maximum number of pages to crawl",
      optional: true,
    },
    crawlEntireDomain: {
      type: "boolean",
      label: "Crawl Entire Domain",
      description: "Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.",
      optional: true,
    },
    allowExternalLinks: {
      type: "boolean",
      label: "Allow External Links",
      description: "Allows the crawler to follow links to external websites",
      optional: true,
    },
    additionalOptions: {
      propDefinition: [
        firecrawl,
        "additionalOptions",
      ],
      description: "Additional parameters to send in the request. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post) for available parameters. Values will be parsed as JSON where applicable. For example, to add the `webhook` param, use the value `{\"webhook\": {\"url\": \"https://your-server-webhook-api.com\",\"headers\": {},\"metadata\": {},\"events\": [\"completed\"]}}`",
    },
  },
  async run({ $ }) {
    const {
      firecrawl, additionalOptions, ...data
    } = this;
    const response = await firecrawl.crawl({
      $,
      data: {
        ...data,
        ...(additionalOptions && parseObjectEntries(additionalOptions)),
      },
    });

    $.export("$summary", `Crawl job started (ID: ${response.id})`);
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
FireCrawl	`firecrawl`	`app`	This component uses the FireCrawl app.
URL	`url`	`string`	The URL to start crawling from
Prompt	`prompt`	`string`	A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents.
Exclude Paths	`excludePaths`	`string[]`	URL pathname regex patterns that exclude matching URLs from the crawl. For example, a value of `blog/.*` for the URL `firecrawl.dev` will exclude any results matching that pattern, such as `https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap`
Include Paths	`includePaths`	`string[]`	Similar to `Exclude Paths`, but if set, only the paths matching the specified patterns will be included
Max Discovery Depth	`maxDiscoveryDepth`	`integer`	Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set sitemap: 'skip', you will only crawl the entered URL and all URLs that are linked on that page.
Sitemap	`sitemap`	`string`	Select a value from the drop down menu:`skipinclude`
Ignore Query Parameters	`ignoreQueryParameters`	`boolean`	Do not re-scrape the same path with different (or none) query parameters
Limit	`limit`	`integer`	Maximum number of pages to crawl
Crawl Entire Domain	`crawlEntireDomain`	`boolean`	Allows the crawler to follow internal links to sibling or parent URLs, not just child paths.
Allow External Links	`allowExternalLinks`	`boolean`	Allows the crawler to follow links to external websites
Additional Options	`additionalOptions`	`object`	Additional parameters to send in the request. See the documentation for available parameters. Values will be parsed as JSON where applicable. For example, to add the `webhook` param, use the value `{"webhook": {"url": "https://your-server-webhook-api.com","headers": {},"metadata": {},"events": ["completed"]}}`

Crawl URL with FireCrawl API on New Team from Microsoft Teams API

Pipedream makes it easy to connect APIs for FireCrawl, Microsoft Teams and 2,800+ other apps remarkably fast.

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

1-24of2,800+apps by most popular

1
-
24
of
2,800+
apps by most popular