Scrape Page with FireCrawl API on New Participant from Demio API

Pipedream makes it easy to connect APIs for FireCrawl, Demio and 2,900+ other apps remarkably fast.

Trigger workflow on

New Participant from the Demio API

Next, do this

Scrape Page with the FireCrawl API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Demio trigger and FireCrawl action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Participant trigger
1. Connect your Demio account
2. Configure timer
3. Select a Event ID
4. Select a Date ID
5. Optional- Select a Status
Configure the Scrape Page action
1. Connect your FireCrawl account
2. Configure URL
3. Optional- Select one or more Formats
4. Optional- Configure Only Main Content
5. Optional- Configure Include Tags
6. Optional- Configure Exclude Tags
7. Optional- Configure Headers
8. Optional- Configure Wait For
9. Optional- Configure Mobile
10. Optional- Configure Timeout
11. Optional- Configure Additional Options
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Participant on Demio

Description:Emit new event for each participant in an event. [See docs here](https://publicdemioapi.docs.apiary.io/#reference/reports/event-date-participants/event-date-participants)

Version:0.0.2

Key:demio-new-participant

View on GitHub

Demio Overview#

Demio, a webinar platform, offers an API that lets you automate various aspects of your webinar experience. With the Demio API on Pipedream, you can seamlessly integrate webinar registration, attendance tracking, and follow-up actions into your business workflows. Pipedream's serverless platform facilitates the creation of complex automations that can trigger actions based on events in Demio, connecting it with hundreds of other apps for a cohesive ecosystem.

Trigger Code#

import demio from "../../demio.app.mjs";
import constants from "../common/constants.mjs";
import { DEFAULT_POLLING_SOURCE_TIMER_INTERVAL } from "@pipedream/platform";

export default {
  key: "demio-new-participant",
  name: "New Participant",
  description: "Emit new event for each participant in an event. [See docs here](https://publicdemioapi.docs.apiary.io/#reference/reports/event-date-participants/event-date-participants)",
  type: "source",
  version: "0.0.2",
  dedupe: "unique",
  props: {
    demio,
    timer: {
      type: "$.interface.timer",
      default: {
        intervalSeconds: DEFAULT_POLLING_SOURCE_TIMER_INTERVAL,
      },
    },
    eventId: {
      propDefinition: [
        demio,
        "eventId",
      ],
    },
    dateId: {
      propDefinition: [
        demio,
        "dateId",
        (c) => ({
          eventId: c.eventId,
        }),
      ],
    },
    status: {
      label: "Status",
      description: "Status of the participant to filter",
      type: "string",
      options: constants.PARTICIPANT_STATUSES,
      optional: true,
    },
  },
  async run({ $ }) {
    let participants = await this.demio.getEventDateParticipants({
      $,
      dateId: this.dateId,
    });

    if (this.status) {
      participants = participants.filter((participant) => participant.status === this.status);
    }

    for (const participant of participants) {
      this.$emit(participant, {
        id: participant.email,
        summary: `New participant ${participant.email}`,
        ts: new Date(),
      });
    }
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Demio	`demio`	`app`	This component uses the Demio app.
	`timer`	`$.interface.timer`
Event ID	`eventId`	`string`	Select a value from the drop down menu.
Date ID	`dateId`	`string`	Select a value from the drop down menu.
Status	`status`	`string`	Select a value from the drop down menu:`attendeddid not attendcompletedleftearlybanned`

Trigger Authentication#

Demio uses API keys for authentication. When you connect your Demio account, Pipedream securely stores the keys so you can easily authenticate to Demio APIs in both code and no-code steps.

To retrieve your API keys,

Navigate to your Demio account and sign in
Go to “Settings” > “API”

About Demio#

Run more effective webinars with less work.

Action#

Scrape Page on FireCrawl

Description:Scrapes a URL and returns content from that page. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/scrape)

Version:1.0.4

Key:firecrawl-scrape-page

View on GitHub

Action Code#

import { OUTPUT_FORMAT_OPTIONS } from "../../common/constants.mjs";
import { parseObjectEntries } from "../../common/utils.mjs";
import firecrawl from "../../firecrawl.app.mjs";

export default {
  key: "firecrawl-scrape-page",
  name: "Scrape Page",
  description:
    "Scrapes a URL and returns content from that page. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/scrape)",
  version: "1.0.4",
  annotations: {
    destructiveHint: false,
    openWorldHint: true,
    readOnlyHint: false,
  },
  type: "action",
  props: {
    firecrawl,
    url: {
      propDefinition: [
        firecrawl,
        "url",
      ],
      description: "The URL to scrape",
    },
    formats: {
      type: "string[]",
      label: "Formats",
      description: "Formats to include in the output",
      options: OUTPUT_FORMAT_OPTIONS,
      optional: true,
    },
    onlyMainContent: {
      type: "boolean",
      label: "Only Main Content",
      description: "Only return the main content of the page, excluding headers, navs, footers, etc.",
      optional: true,
    },
    includeTags: {
      type: "string[]",
      label: "Include Tags",
      description: "Tags to include in the output",
      optional: true,
    },
    excludeTags: {
      type: "string[]",
      label: "Exclude Tags",
      description: "Tags to exclude from the output",
      optional: true,
    },
    headers: {
      type: "object",
      label: "Headers",
      description: "Headers to send with the request. Can be used to send cookies, user-agent, etc.",
      optional: true,
    },
    waitFor: {
      type: "integer",
      label: "Wait For",
      description: "Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load.",
      optional: true,
    },
    mobile: {
      type: "boolean",
      label: "Mobile",
      description: "Set to `true` to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots",
      optional: true,
    },
    timeout: {
      type: "integer",
      label: "Timeout",
      description: "Timeout in milliseconds for the request",
      optional: true,
    },
    additionalOptions: {
      propDefinition: [
        firecrawl,
        "additionalOptions",
      ],
    },
  },
  async run({ $ }) {
    const {
      firecrawl, additionalOptions, ...data
    } = this;
    const response = await firecrawl.scrape({
      $,
      data: {
        ...data,
        ...(additionalOptions && parseObjectEntries(additionalOptions)),
      },
    });

    $.export("$summary", `Successfully scraped content from ${this.url}`);
    return response;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
FireCrawl	`firecrawl`	`app`	This component uses the FireCrawl app.
URL	`url`	`string`	The URL to scrape
Formats	`formats`	`string[]`	Select a value from the drop down menu:`{ "value": "markdown", "label": "Markdown" }{ "value": "html", "label": "HTML" }{ "value": "rawHtml", "label": "Raw HTML" }{ "value": "links", "label": "Links" }{ "value": "screenshot", "label": "Screenshot" }{ "value": "screenshot@fullPage", "label": "Full Page Screenshot" }{ "value": "json", "label": "JSON" }`
Only Main Content	`onlyMainContent`	`boolean`	Only return the main content of the page, excluding headers, navs, footers, etc.
Include Tags	`includeTags`	`string[]`	Tags to include in the output
Exclude Tags	`excludeTags`	`string[]`	Tags to exclude from the output
Headers	`headers`	`object`	Headers to send with the request. Can be used to send cookies, user-agent, etc.
Wait For	`waitFor`	`integer`	Specify a delay in milliseconds before fetching the content, allowing the page sufficient time to load.
Mobile	`mobile`	`boolean`	Set to `true` to emulate scraping from a mobile device. Useful for testing responsive pages and taking mobile screenshots
Timeout	`timeout`	`integer`	Timeout in milliseconds for the request
Additional Options	`additionalOptions`	`object`	Additional parameters to send in the request. See the documentation for available parameters. Values will be parsed as JSON where applicable.