# Scrape URL — Decodo

> Fetch the live contents of a web page through Decodo's scraping proxies. Use when you need the current content of a URL — a public page, a search-engine results page via a target template (e.g. google_search with a query), or a…

- Key: `decodo-scrape-url`
- Type: Action (Write)
- Version: 0.0.1
- App: Decodo (`decodo`) — https://pipedream.com/apps/decodo.md
- This page (HTML): https://pipedream.com/apps/decodo/actions/scrape-url
- Hints: open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/decodo/actions/scrape-url/scrape-url.mjs

## Description

Fetch the live contents of a web page through Decodo's scraping proxies. Use when you need the current content of a URL — a public page, a search-engine results page via a `target` template (e.g. `google_search` with a `query`), or a JavaScript-heavy page (set `headless` to `html` to render it first). Returns an array of results, each with the page `content`, the HTTP `status_code`, response `headers`, and `cookies`. Example: to read `https://example.com` as clean Markdown, set `url` to `https://example.com` and `markdown` to `true`; the returned `content` is the page converted to Markdown (`# Example Domain ...`). [See the documentation](https://help.decodo.com/docs/web-scraping-api-parameters)

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `url` | `string` | No | The URL to scrape. Required unless using a target template that accepts query instead. E.g. https://ip.decodo.com |
| `query` | `string` | No | Search query for target templates that support it (instead of or in addition to url). E.g. pizza near me |
| `target` | `string` | No | Target template for specialized scrapers. E.g. google_search. See the documentation |
| `proxyPool` | `string` | No | standard handles simple pages; premium resolves complex anti-bot measures and is used for target templates. Defaults to premium. |
| `headless` | `string` | No | html enables JavaScript rendering. png enables a screenshot response. Some target templates force JS rendering by default. See the documentation |
| `geo` | `string` | No | The geographical location for the request. Defaults to auto-randomized. E.g. United States |
| `domain` | `string` | No | Top-level domain for localized results. E.g. com, co.uk, fr. Defaults to com. |
| `locale` | `string` | No | Changes the search page web interface language (not the results). Matched with the domain parameter by default. E.g. en-US, en-GB |
| `headers` | `object` | No | Additional request headers to send to the target. By default these are not forwarded unless Force Headers is enabled. E.g. {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"} |
| `cookies` | `object` | No | Cookies to send with the request, e.g. to access pages as a logged-in user, as a map of cookie name to value (values are sent as strings). By default these are not forwarded unless Force Cookies is enabled. E.g. {"sessionid": "8e1f3b56-7abc-47e6-b2e2-1d273b0a1c4d", "logged_in": "true", "locale": "en-US"} |
| `forceCookies` | `boolean` | No | By default, user-provided cookies are not forwarded to the target. Set to true to override this behaviour. |
| `forceHeaders` | `boolean` | No | By default, user-provided headers are not forwarded to the target. Set to true to override this behaviour. |
| `deviceType` | `string` | No | Device type and browser for the request. Defaults to desktop. See the documentation |
| `parse` | `boolean` | No | For certain target templates, set to true to retrieve structured/parsed data instead of raw HTML. |
| `sessionId` | `string` | No | Reuse the same IP for multiple requests for up to 10 minutes. E.g. 1234 |
| `httpMethod` | `string` | No | HTTP method for the target request. Defaults to GET. Use POST with a base64-encoded Payload. |
| `payload` | `string` | No | Base64-encoded POST request body. Used when HTTP Method is POST. E.g. eyJmb28iOiJiYXIifQ== for {"foo":"bar"}. |
| `successfulStatusCodes` | `integer[]` | No | HTTP response codes to treat as successful so content is still returned. E.g. [401, 404] |
| `markdown` | `boolean` | No | Parse HTML output into Markdown. Useful for reducing tokens when feeding results into LLM models. |
| `xhr` | `boolean` | No | When enabled, retrieves a list of XHR and fetch requests made by the page. |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "decodo",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "decodo-scrape-url",
  arguments: {
    url: "URL",
    query: "Query",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "decodo-scrape-url",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    decodo: { authProvisionId: "apn_xxxxxxx" },
    url: "URL",
    query: "Query",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "decodo-scrape-url",
    "configured_props": {
      "decodo": { "authProvisionId": "apn_xxxxxxx" },
      "url": "URL",
      "query": "Query"
    }
  }'
```

---

- App: https://pipedream.com/apps/decodo.md · All apps: https://pipedream.com/apps