Crawl URL with FireCrawl API on New Custom Event from Strava API

Pipedream makes it easy to connect APIs for FireCrawl, Strava and 2,700+ other apps remarkably fast.

Trigger workflow on

New Custom Event from the Strava API

Next, do this

Crawl URL with the FireCrawl API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Strava trigger and FireCrawl action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Custom Event trigger
1. Connect your Strava account
2. Select one or more Strava Events
3. Configure stravaApphook
Configure the Crawl URL action
1. Connect your FireCrawl account
2. Configure URL
3. Optional- Configure Exclude Paths
4. Optional- Configure Include Paths
5. Optional- Configure Max Depth
6. Optional- Configure Ignore Sitemap
7. Optional- Configure Ignore Query Parameters
8. Optional- Configure Limit
9. Optional- Configure Allow Backward Links
10. Optional- Configure Allow External Links
11. Optional- Configure Additional Options
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Custom Event on Strava

Description:Emit new event when an activity is created, updated, or deleted

Version:0.0.3

Key:strava-custom-events

View on GitHub

Strava Overview#

The Strava API lets you tap into the robust data from Strava's fitness app, which is a playground for athletes worldwide to track their workouts. With Pipedream, you can automate actions based on activities uploaded to Strava, such as running, biking, or swimming. Imagine syncing workout data to spreadsheets for analysis, auto-posting achievements to social media, or integrating with calendar apps for better scheduling. Pipedream's serverless platform makes it a breeze to create workflows that can listen for Strava webhooks, process data, and trigger actions in countless other apps.

Trigger Code#

import strava from "../../strava.app.mjs";

export default {
  key: "strava-custom-events",
  name: "New Custom Event",
  description: "Emit new event when an activity is created, updated, or deleted",
  version: "0.0.3",
  type: "source",
  props: {
    strava,
    eventNameOptions: {
      label: "Strava Events",
      description: "Select from events",
      type: "string[]",
      async options() {
        return [
          "activity.create",
          "activity.update",
          "activity.delete",
        ];
      },
    },
    stravaApphook: {
      type: "$.interface.apphook",
      appProp: "strava",
      async eventNames() {
        return this.eventNameOptions;
      },
    },
  },
  async run(event) {
    console.log(event);
    const ts = event.event_time * 1000;
    if (event.object_type === "activity") {
      let details;
      // Optimistically fetch activity details. When an event is deleted, this will fail
      try {
        details = await this.strava.getActivity({
          activityId: event.object_id,
        });
      } catch (err) {
        console.log(`Error fetching activity details: ${err}`);
      }
      this.$emit(
        {
          event,
          details,
        },
        {
          summary: `Activity ${event.aspect_type}d`,
          ts,
        },
      );
    }
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Strava	`strava`	`app`	This component uses the Strava app.
Strava Events	`eventNameOptions`	`string[]`	Select a value from the drop down menu.
	`stravaApphook`	`$.interface.apphook`

Trigger Authentication#

Strava uses OAuth authentication. When you connect your Strava account, Pipedream will open a popup window where you can sign into Strava and grant Pipedream permission to connect to your account. Pipedream securely stores and automatically refreshes the OAuth tokens so you can easily authenticate any Strava API.

Pipedream requests the following authorization scopes when you connect your account:

readactivity:writeactivity:readactivity:read_all

About Strava#

Designed by athletes, for athletes, Strava's mobile app and website connect millions of runners and cyclists through the sports they love.

Action#

Crawl URL on FireCrawl

Description:Crawls a given URL and returns the contents of sub-pages. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post)

Version:1.0.3

Key:firecrawl-crawl-url

View on GitHub

Action Code#

import { parseObjectEntries } from "../../common/utils.mjs";
import firecrawl from "../../firecrawl.app.mjs";

export default {
  key: "firecrawl-crawl-url",
  name: "Crawl URL",
  description: "Crawls a given URL and returns the contents of sub-pages. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post)",
  version: "1.0.3",
  type: "action",
  props: {
    firecrawl,
    url: {
      propDefinition: [
        firecrawl,
        "url",
      ],
    },
    excludePaths: {
      type: "string[]",
      label: "Exclude Paths",
      description: "URL pathname regex patterns that exclude matching URLs from the crawl. For example, a value of `blog/.*` for the URL `firecrawl.dev` will exclude any results matching that pattern, such as `https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap`",
      optional: true,
    },
    includePaths: {
      type: "string[]",
      label: "Include Paths",
      description: "Similar to `Exclude Paths`, but if set, only the paths matching the specified patterns will be included",
      optional: true,
    },
    maxDepth: {
      type: "integer",
      label: "Max Depth",
      description: "Maximum depth to crawl relative to the entered URL",
      optional: true,
    },
    ignoreSitemap: {
      type: "boolean",
      label: "Ignore Sitemap",
      description: "Ignore the website sitemap when crawling",
      optional: true,
    },
    ignoreQueryParameters: {
      type: "boolean",
      label: "Ignore Query Parameters",
      description: "Do not re-scrape the same path with different (or none) query parameters",
      optional: true,
    },
    limit: {
      type: "integer",
      label: "Limit",
      description: "Maximum number of pages to crawl",
      optional: true,
    },
    allowBackwardLinks: {
      type: "boolean",
      label: "Allow Backward Links",
      description: "Enables the crawler to navigate from a specific URL to previously linked pages",
      optional: true,
    },
    allowExternalLinks: {
      type: "boolean",
      label: "Allow External Links",
      description: "Allows the crawler to follow links to external websites",
      optional: true,
    },
    additionalOptions: {
      propDefinition: [
        firecrawl,
        "additionalOptions",
      ],
      description: "Additional parameters to send in the request. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post) for available parameters. Values will be parsed as JSON where applicable. For example, to add the `webhook` param, use the value `{\"webhook\": {\"url\": \"https://your-server-webhook-api.com\",\"headers\": {},\"metadata\": {},\"events\": [\"completed\"]}}`",
    },
  },
  async run({ $ }) {
    const {
      firecrawl, additionalOptions, ...data
    } = this;
    const response = await firecrawl.crawl({
      $,
      data: {
        ...data,
        ...(additionalOptions && parseObjectEntries(additionalOptions)),
      },
    });

    $.export("$summary", `Crawl job started (ID: ${response.id})`);
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
FireCrawl	`firecrawl`	`app`	This component uses the FireCrawl app.
URL	`url`	`string`	The URL to start crawling from
Exclude Paths	`excludePaths`	`string[]`	URL pathname regex patterns that exclude matching URLs from the crawl. For example, a value of `blog/.*` for the URL `firecrawl.dev` will exclude any results matching that pattern, such as `https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap`
Include Paths	`includePaths`	`string[]`	Similar to `Exclude Paths`, but if set, only the paths matching the specified patterns will be included
Max Depth	`maxDepth`	`integer`	Maximum depth to crawl relative to the entered URL
Ignore Sitemap	`ignoreSitemap`	`boolean`	Ignore the website sitemap when crawling
Ignore Query Parameters	`ignoreQueryParameters`	`boolean`	Do not re-scrape the same path with different (or none) query parameters
Limit	`limit`	`integer`	Maximum number of pages to crawl
Allow Backward Links	`allowBackwardLinks`	`boolean`	Enables the crawler to navigate from a specific URL to previously linked pages
Allow External Links	`allowExternalLinks`	`boolean`	Allows the crawler to follow links to external websites
Additional Options	`additionalOptions`	`object`	Additional parameters to send in the request. See the documentation for available parameters. Values will be parsed as JSON where applicable. For example, to add the `webhook` param, use the value `{"webhook": {"url": "https://your-server-webhook-api.com","headers": {},"metadata": {},"events": ["completed"]}}`