# AI Data Extraction — Scrapfly

> Automate content extraction from any text-based source using AI, LLM, and custom parsing. See the documentation

- Key: `scrapfly-ai-data-extraction`
- Type: Action (Read-only)
- Version: 0.1.2
- App: Scrapfly (`scrapfly`) — https://pipedream.com/apps/scrapfly.md
- This page (HTML): https://pipedream.com/apps/scrapfly/actions/ai-data-extraction
- Hints: read-only · open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/scrapfly/actions/ai-data-extraction/ai-data-extraction.mjs

## Description

Automate content extraction from any text-based source using AI, LLM, and custom parsing. [See the documentation](https://scrapfly.io/docs/extraction-api/getting-started)

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `body` | `string` | Yes | The file containing the content of the page you want to extract data from. The content must be in the format specified by Content Type. Provide either a file URL or a path to a file in the /tmp directory (for example, /tmp/myFile.txt) |
| `contentType` | `string` | Yes | Content type of the document pass in the body - You must specify the content type of the document by using this parameter or via the content-type header. This parameter has priority over the content-type header. |
| `url` | `string` | Yes | This URL is used to transform any relative URLs in the document into absolute URLs automatically. It can be either the base URL or the exact URL of the document. Must be url encoded. |
| `charset` | `string` | No | Charset of the document pass in the body. If you are not sure, you can use the auto value and we will try to detect it. Bad charset can lead to bad extraction, so it's important to set it correctly. The most common charset is utf-8 for text document and ascii for binary. The symptom of a bad charset is that the text is not correctly displayed (accent, special characters, etc). |
| `extractionTemplate` | `string` | No | Define an extraction template to get structured data. Use an ephemeral template (declared on the fly on the API call) or a stored template (declared in the dashboard) by using the template name. |
| `extractionPrompt` | `string` | No | Instruction to extract data or ask a question on the scraped content with an LLM (Large Language Model). Must be url encoded. |
| `extractionModel` | `string` | No | AI Extraction to auto parse document to get structured data. E.g., product, review, real-estate, article. |
| `webhookName` | `string` | No | Queue you scrape request and redirect API response to a provided webhook endpoint. You can create a webhook endpoint from your dashboard, it takes the name of the webhook. Webhooks are scoped to the given project/env. |
| `syncDir` | `dir` | No | SyncDir |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "scrapfly",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "scrapfly-ai-data-extraction",
  arguments: {
    body: "File Path or URL",
    contentType: "Content Type",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "scrapfly-ai-data-extraction",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    scrapfly: { authProvisionId: "apn_xxxxxxx" },
    body: "File Path or URL",
    contentType: "Content Type",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "scrapfly-ai-data-extraction",
    "configured_props": {
      "scrapfly": { "authProvisionId": "apn_xxxxxxx" },
      "body": "File Path or URL",
      "contentType": "Content Type"
    }
  }'
```

---

- App: https://pipedream.com/apps/scrapfly.md · All apps: https://pipedream.com/apps