# Scrape Website HTML — WebScraping.AI

> Returns the full HTML content of a webpage specified by the URL. See the documentation:

- Key: `webscraping_ai-scrape-website-html`
- Type: Action (Write)
- Version: 0.0.2
- App: WebScraping.AI (`webscraping_ai`) — https://pipedream.com/apps/webscraping-ai.md
- This page (HTML): https://pipedream.com/apps/webscraping-ai/actions/scrape-website-html
- Hints: open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/webscraping_ai/actions/scrape-website-html/scrape-website-html.mjs

## Description

Returns the full HTML content of a webpage specified by the URL. [See the documentation](https://webscraping.ai/docs#tag/HTML/operation/getHTML):

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `targetUrl` | `string` | Yes | The URL of the webpage to scrape. |
| `headers` | `object` | No | HTTP headers to pass to the target page |
| `timeout` | `integer` | No | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000). |
| `js` | `boolean` | No | Execute on-page JavaScript using a headless browser (true by default) |
| `jsTimeout` | `integer` | No | Maximum JavaScript rendering time in ms. Default: 2000 |
| `waitFor` | `string` | No | CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. |
| `proxy` | `string` | No | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details. |
| `country` | `string` | No | Country of the proxy to use (us by default) |
| `customProxy` | `string` | No | Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (Smartproxy for example). |
| `device` | `string` | No | Type of device emulation. Default is desktop |
| `errorOn404` | `boolean` | No | Return error on 404 HTTP status on the target page (false by default) |
| `errorOnRedirect` | `boolean` | No | Return error on redirect on the target page (false by default) |
| `jsScript` | `string` | No | Custom JavaScript code to execute on the target page. Example: document.querySelector('button').click(); |
| `format` | `string` | No | Format of the response (text by default). json will return a JSON object with the response, text will return a plain text/HTML response. |
| `returnScriptResult` | `boolean` | No | Return result of the custom JavaScript code (js_script parameter) execution on the target page (false by default, page HTML will be returned). |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "webscraping_ai",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "webscraping_ai-scrape-website-html",
  arguments: {
    targetUrl: "Target URL",
    headers: "Headers",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "webscraping_ai-scrape-website-html",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    webscraping_ai: { authProvisionId: "apn_xxxxxxx" },
    targetUrl: "Target URL",
    headers: "Headers",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "webscraping_ai-scrape-website-html",
    "configured_props": {
      "webscraping_ai": { "authProvisionId": "apn_xxxxxxx" },
      "targetUrl": "Target URL",
      "headers": "Headers"
    }
  }'
```

---

- App: https://pipedream.com/apps/webscraping-ai.md · All apps: https://pipedream.com/apps