# Crawl URL — FireCrawl

> Crawls a given URL and returns the contents of sub-pages. See the documentation

- Key: `firecrawl-crawl-url`
- Type: Action (Write)
- Version: 1.1.2
- App: FireCrawl (`firecrawl`) — https://pipedream.com/apps/firecrawl.md
- This page (HTML): https://pipedream.com/apps/firecrawl/actions/crawl-url
- Hints: open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/firecrawl/actions/crawl-url/crawl-url.mjs

## Description

Crawls a given URL and returns the contents of sub-pages. [See the documentation](https://docs.firecrawl.dev/api-reference/endpoint/crawl-post)

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `url` | `string` | Yes | The URL to start crawling from |
| `prompt` | `string` | No | A prompt to use to generate the crawler options (all the parameters below) from natural language. Explicitly set parameters will override the generated equivalents. |
| `excludePaths` | `string[]` | No | URL pathname regex patterns that exclude matching URLs from the crawl. For example, a value of blog/.* for the URL firecrawl.dev will exclude any results matching that pattern, such as https://www.firecrawl.dev/blog/firecrawl-launch-week-1-recap |
| `includePaths` | `string[]` | No | Similar to Exclude Paths, but if set, only the paths matching the specified patterns will be included |
| `maxDiscoveryDepth` | `integer` | No | Maximum depth to crawl based on discovery order. The root site and sitemapped pages has a discovery depth of 0. For example, if you set it to 1, and you set sitemap: 'skip', you will only crawl the entered URL and all URLs that are linked on that page. |
| `sitemap` | `string` | No | Sitemap mode when crawling. If you set it to 'skip', the crawler will ignore the website sitemap and only crawl the entered URL and discover pages from there onwards. |
| `ignoreQueryParameters` | `boolean` | No | Do not re-scrape the same path with different (or none) query parameters |
| `limit` | `integer` | No | Maximum number of pages to crawl |
| `crawlEntireDomain` | `boolean` | No | Allows the crawler to follow internal links to sibling or parent URLs, not just child paths. |
| `allowExternalLinks` | `boolean` | No | Allows the crawler to follow links to external websites |
| `additionalOptions` | `object` | No | Additional parameters to send in the request. See the documentation for available parameters. Values will be parsed as JSON where applicable. For example, to add the webhook param, use the value {"webhook": {"url": "https://your-server-webhook-api.com","headers": {},"metadata": {},"events": ["completed"]}} |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "firecrawl",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "firecrawl-crawl-url",
  arguments: {
    url: "URL",
    prompt: "Prompt",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "firecrawl-crawl-url",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    firecrawl: { authProvisionId: "apn_xxxxxxx" },
    url: "URL",
    prompt: "Prompt",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "firecrawl-crawl-url",
    "configured_props": {
      "firecrawl": { "authProvisionId": "apn_xxxxxxx" },
      "url": "URL",
      "prompt": "Prompt"
    }
  }'
```

---

- App: https://pipedream.com/apps/firecrawl.md · All apps: https://pipedream.com/apps