# Extract Data From URL — Automatic Data Extraction

> Extract data from a specified URL See the docs here

- Key: `automatic_data_extraction-extract-data-from-url`
- Type: Action (Write)
- Version: 0.0.3
- App: Automatic Data Extraction (`automatic_data_extraction`) — https://pipedream.com/apps/automatic-data-extraction.md
- This page (HTML): https://pipedream.com/apps/automatic-data-extraction/actions/extract-data-from-url
- Hints: open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/automatic_data_extraction/actions/extract-data-from-url/extract-data-from-url.mjs

## Description

Extract data from a specified URL [See the docs here](https://docs.zyte.com/automatic-extraction-integrations.html#node-js)

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `url` | `string` | Yes | URL of web page to extract from. Must be a valid http:// or https:// URL. |
| `pageType` | `string` | Yes | Type of extraction to perform. |
| `meta` | `string` | No | User UTF-8 string, which will be passed through the extraction pipeline and returned in the query result. Max size 4 Kb. |
| `articleBodyRaw` | `boolean` | No | Whether or not to include article HTML in article extractions. True by default. Setting this to false can reduce response size significantly if HTML is not required. |
| `fullHtml` | `boolean` | No | Include the full, raw HTML of the target web page in the query result. This is a premium feature that is disabled by default. Open a support ticket if you wish to have it enabled for your account. |
| `customHtml` | `string` | No | HTML source to be scraped. Extraction will be done from the provided HTML with additional resources (images, CSS, etc.) downloaded from the provided url. JavaScript processing will be disabled. The String should be UTF-8 encoded. The maximum length is 2,000,000 characters, longer requests will be rejected. |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "automatic_data_extraction",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "automatic_data_extraction-extract-data-from-url",
  arguments: {
    url: "URL",
    pageType: "Page Type",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "automatic_data_extraction-extract-data-from-url",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    automatic_data_extraction: { authProvisionId: "apn_xxxxxxx" },
    url: "URL",
    pageType: "Page Type",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "automatic_data_extraction-extract-data-from-url",
    "configured_props": {
      "automatic_data_extraction": { "authProvisionId": "apn_xxxxxxx" },
      "url": "URL",
      "pageType": "Page Type"
    }
  }'
```

---

- App: https://pipedream.com/apps/automatic-data-extraction.md · All apps: https://pipedream.com/apps