Scrape New Page with Spider API on New Session Created from Zep API

Pipedream makes it easy to connect APIs for Spider, Zep and 2,700+ other apps remarkably fast.

Trigger workflow on

New Session Created from the Zep API

Next, do this

Scrape New Page with the Spider API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

This integration creates a workflow with a Zep trigger and Spider action. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Select this integration
Configure the New Session Created trigger
1. Connect your Zep account
2. Configure timer
Configure the Scrape New Page action
1. Connect your Spider account
2. Configure infoBox
3. Configure URL
4. Optional- Configure Limit
5. Optional- Configure Store Data
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Details#

This integration uses pre-built, source-available components from Pipedream's GitHub repo. These components are developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Trigger#

New Session Created on Zep

Description:Emit new event when a new session is created in Zep. [See the documentation](https://help.getzep.com/api-reference/memory/list-sessions)

Version:0.0.1

Key:zep-new-session

View on GitHub

Trigger Code#

import common from "../common/base.mjs";

export default {
  ...common,
  key: "zep-new-session",
  name: "New Session Created",
  description: "Emit new event when a new session is created in Zep. [See the documentation](https://help.getzep.com/api-reference/memory/list-sessions)",
  version: "0.0.1",
  type: "source",
  dedupe: "unique",
  methods: {
    ...common.methods,
    getNewResults(lastTs, max) {
      return this.getSessions({
        lastTs,
        max,
        orderBy: "created_at",
      });
    },
    generateMeta(session) {
      return {
        id: session.session_id,
        summary: `New Session: ${session.session_id}`,
        ts: Date.parse(session.created_at),
      };
    },
  },
};

Trigger Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Zep	`zep`	`app`	This component uses the Zep app.
N/A	`db`	`$.service.db`	This component uses `$.service.db` to maintain state between executions.
	`timer`	`$.interface.timer`

Trigger Authentication#

Zep uses API keys for authentication. When you connect your Zep account, Pipedream securely stores the keys so you can easily authenticate to Zep APIs in both code and no-code steps.

About Zep#

The Foundational Memory Layer for AI. Equip your agents with the knowledge to complete tasks, from the mundane to monumental.

Action#

Scrape New Page on Spider

Description:Initiates a new page scrape (crawl). [See the documentation](https://spider.cloud/docs/api#crawl-website)

Version:0.0.1

Key:spider-scrape-new-page

View on GitHub

Action Code#

import spider from "../../spider.app.mjs";

export default {
  key: "spider-scrape-new-page",
  name: "Scrape New Page",
  description: "Initiates a new page scrape (crawl). [See the documentation](https://spider.cloud/docs/api#crawl-website)",
  version: "0.0.1",
  type: "action",
  props: {
    spider,
    infoBox: {
      type: "alert",
      alertType: "info",
      content: "See [the Spider documentation](https://spider.cloud/docs/api#crawl-website) for information on limits and best practices.",
    },
    url: {
      type: "string",
      label: "URL",
      description: "The URI resource to crawl, e.g. `https://spider.cloud`. This can be a comma split list for multiple urls.",
    },
    limit: {
      type: "integer",
      label: "Limit",
      description: "The maximum amount of pages allowed to crawl per website. Default is 0, which crawls all pages.",
      optional: true,
    },
    storeData: {
      type: "boolean",
      label: "Store Data",
      description: "Decide whether to store data. Default is `false`.",
      optional: true,
    },
  },
  async run({ $ }) {
    const content = await this.spider.initiateCrawl({
      $,
      data: {
        url: this.url,
        limit: this.limit,
        store_data: this.storeData,
      },
    });
    $.export("$summary", `Successfully scraped URL ${this.url}`);
    return content;
  },
};

Action Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI.

Label	Prop	Type	Description
Spider	`spider`	`app`	This component uses the Spider app.
URL	`url`	`string`	The URI resource to crawl, e.g. `https://spider.cloud`. This can be a comma split list for multiple urls.
Limit	`limit`	`integer`	The maximum amount of pages allowed to crawl per website. Default is 0, which crawls all pages.
Store Data	`storeData`	`boolean`	Decide whether to store data. Default is `false`.