# Scrape Website Text — WebScraping.AI

> Returns the visible text content of a webpage specified by the URL. See the documentation.

- Key: `webscraping_ai-scrape-website-text`
- Type: Action (Write)
- Version: 0.0.2
- App: WebScraping.AI (`webscraping_ai`) — https://pipedream.com/apps/webscraping-ai.md
- This page (HTML): https://pipedream.com/apps/webscraping-ai/actions/scrape-website-text
- Hints: open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/webscraping_ai/actions/scrape-website-text/scrape-website-text.mjs

## Description

Returns the visible text content of a webpage specified by the URL. [See the documentation](https://webscraping.ai/docs#tag/Text/operation/getText).

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `targetUrl` | `string` | Yes | The URL of the webpage to scrape. |
| `headers` | `object` | No | HTTP headers to pass to the target page |
| `timeout` | `integer` | No | Maximum web page retrieval time in ms. Increase it in case of timeout errors (10000 by default, maximum is 30000). |
| `js` | `boolean` | No | Execute on-page JavaScript using a headless browser (true by default) |
| `jsTimeout` | `integer` | No | Maximum JavaScript rendering time in ms. Default: 2000 |
| `waitFor` | `string` | No | CSS selector to wait for before returning the page content. Useful for pages with dynamic content loading. Overrides js_timeout. |
| `proxy` | `string` | No | Type of proxy, use residential proxies if your site restricts traffic from datacenters (datacenter by default). Note that residential proxy requests are more expensive than datacenter, see the pricing page for details. |
| `country` | `string` | No | Country of the proxy to use (us by default) |
| `customProxy` | `string` | No | Your own proxy URL to use instead of our built-in proxy pool in "http://user:password@host:port" format (Smartproxy for example). |
| `device` | `string` | No | Type of device emulation. Default is desktop |
| `errorOn404` | `boolean` | No | Return error on 404 HTTP status on the target page (false by default) |
| `errorOnRedirect` | `boolean` | No | Return error on redirect on the target page (false by default) |
| `jsScript` | `string` | No | Custom JavaScript code to execute on the target page. Example: document.querySelector('button').click(); |
| `textFormat` | `string` | No | The format of the returned text content. Default: json |
| `returnLinks` | `boolean` | No | Whether to include links in the returned text content. Works only when Text Format is json. |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "webscraping_ai",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "webscraping_ai-scrape-website-text",
  arguments: {
    targetUrl: "Target URL",
    headers: "Headers",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "webscraping_ai-scrape-website-text",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    webscraping_ai: { authProvisionId: "apn_xxxxxxx" },
    targetUrl: "Target URL",
    headers: "Headers",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "webscraping_ai-scrape-website-text",
    "configured_props": {
      "webscraping_ai": { "authProvisionId": "apn_xxxxxxx" },
      "targetUrl": "Target URL",
      "headers": "Headers"
    }
  }'
```

---

- App: https://pipedream.com/apps/webscraping-ai.md · All apps: https://pipedream.com/apps