# Scrape with JS Rendering — ScrapeNinja

> Uses the ScrapeNinja real Chrome browser engine to scrape pages that require JS rendering. See the documentation

- Key: `scrapeninja-scrape-with-js-rendering`
- Type: Action (Write)
- Version: 0.0.2
- App: ScrapeNinja (`scrapeninja`) — https://pipedream.com/apps/scrapeninja.md
- This page (HTML): https://pipedream.com/apps/scrapeninja/actions/scrape-with-js-rendering
- Hints: open-world
- Source: https://github.com/PipedreamHQ/pipedream/blob/master/components/scrapeninja/actions/scrape-with-js-rendering/scrape-with-js-rendering.mjs

## Description

Uses the ScrapeNinja real Chrome browser engine to scrape pages that require JS rendering. [See the documentation](https://scrapeninja.net/docs/api-reference/scrape-js/)

## Props

| Prop | Type | Required | Description |
|---|---|---|---|
| `url` | `string` | Yes | The URL to scrape. |
| `waitForSelector` | `string` | No | CSS selector to wait for before considering the page loaded. |
| `postWaitTime` | `integer` | No | Wait for specified amount of seconds after page load (from 1 to 12s). Use this only if ScrapeNinja failed to wait for required page elements automatically. |
| `dumpIframe` | `string` | No | If some particular iframe needs to be dumped, specify its name HTML value in this argument. The ScrapeNinja JS renderer will wait for CSS selector to wait for iframe DOM elements to appear inside. |
| `waitForSelectorIframe` | `string` | No | If Dump Iframe is activated, this property allows to wait for CSS selector inside this iframe. |
| `extractorTargetIframe` | `boolean` | No | If Dump Iframe is activated, this property allows to run JS extractor function against iframe HTML instead of running it against base body. This is only useful if Dump Iframe is activated. |
| `headers` | `string[]` | No | Custom headers to send with the request. By default, regular Chrome browser headers are sent to the target URL. |
| `retryNum` | `integer` | No | Amount of attempts. |
| `geo` | `string` | No | Geo location for basic proxy pools (you can purchase premium ScrapeNinja proxies for wider country selection and higher proxy quality). Read more about ScrapeNinja proxy setup |
| `proxy` | `string` | No | Premium or your own proxy URL (overrides Geo prop). Read more about ScrapeNinja proxy setup. |
| `timeout` | `integer` | No | Timeout per attempt, in seconds. Each retry will take [timeout] number of seconds. |
| `textNotExpected` | `string[]` | No | Text which will trigger a retry from another proxy address. |
| `statusNotExpected` | `integer[]` | No | HTTP response statuses which will trigger a retry from another proxy address. |
| `blockImages` | `boolean` | No | Block images from loading. This will speed up page loading and reduce bandwidth usage. |
| `blockMedia` | `boolean` | No | Block (CSS, fonts) from loading. This will speed up page loading and reduce bandwidth usage. |
| `screenshot` | `boolean` | No | Take a screenshot of the page. Pass "false" to increase the speed of the request. |
| `catchAjaxHeadersUrlMask` | `string` | No | Useful to dump some XHR response. Pass URL mask here. For example, if you need to catch all requests to https://example.com/api/data.json, pass "api/data.json" here. In response, you will get new property .info.catchedAjax with the XHR response data - { url, method, headers[], body , status, responseHeaders{} } |
| `viewportWidth` | `integer` | No | Width of the viewport. |
| `viewportHeight` | `integer` | No | Height of the viewport. |
| `viewportDeviceScaleFactor` | `integer` | No | Device scale factor for the viewport. |
| `viewportHasTouch` | `boolean` | No | Whether the viewport has touch capabilities. |
| `viewportIsMobile` | `boolean` | No | Whether the viewport is mobile. |
| `viewportIsLandscape` | `boolean` | No | Whether the viewport is in landscape mode. |
| `extractor` | `string` | No | Custom JS function to extract JSON values from scraped HTML. Write&test your own extractor on https://scrapeninja.net/cheerio-sandbox/ |

## Run it

**MCP**

```ts
import { Client } from "@modelcontextprotocol/sdk/client/index.js"
import { StreamableHTTPClientTransport } from "@modelcontextprotocol/sdk/client/streamableHttp.js"
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const accessToken = await pd.rawAccessToken

const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": process.env.PIPEDREAM_PROJECT_ID!,
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "scrapeninja",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// listTools() hands your model this tool's input schema, so it can
// fill the arguments itself:
const result = await mcp.callTool({
  name: "scrapeninja-scrape-with-js-rendering",
  arguments: {
    url: "URL",
    waitForSelector: "Wait For Selector",
  },
})
```

**TypeScript**

```ts
import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "scrapeninja-scrape-with-js-rendering",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    scrapeninja: { authProvisionId: "apn_xxxxxxx" },
    url: "URL",
    waitForSelector: "Wait For Selector",
  },
})

console.log(result)
```

**cURL**

```bash
curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "scrapeninja-scrape-with-js-rendering",
    "configured_props": {
      "scrapeninja": { "authProvisionId": "apn_xxxxxxx" },
      "url": "URL",
      "waitForSelector": "Wait For Selector"
    }
  }'
```

---

- App: https://pipedream.com/apps/scrapeninja.md · All apps: https://pipedream.com/apps