View as Markdown
Piloterr icon

Piloterr ACTION

Get Website Crawler

Obtains HTML from a given website through web scraping for high performance access and interpretation. See the documentation
  • Action
  • Read only
  • API key
  • SDK
  • MCP

IMPLEMENTATION

Call this tool

Connect a user's Piloterr account once, then configure and run Get Website Crawler from your backend or agent.

import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "piloterr-get-website-crawler",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    piloterr: { authProvisionId: "apn_xxxxxxx" },
    url: "Website URL",
    impersonateVersion: "Impersonate Version",
  },
})

console.log(result)

SCHEMA

Inputs

Pipedream supplies the connected account. Your application provides the operation-specific values below. Dynamic inputs are resolved against that user's account.

Get Website Crawler inputs
Property Type Description
url Website URL string
The URL of the website to obtain HTML from
Required
impersonateVersion Impersonate Version string
Impersonate a browser version
Optional
allowRedirects Allow Redirects boolean
If set to false, do not follow redirects. true by default.
Optional
returnPageSource Return Page Source boolean
If set to false, the response will be a JSON object with the response body of the page. true by default.
Optional

REFERENCE

Tool details

Behavior hints are published with the component in the Pipedream registry and surface as MCP tool annotations, so an agent can reason about a tool before it calls it.

Registry key
piloterr-get-website-crawler
Version
0.0.2
App
Piloterr
Authentication
API key
Read-only
Yes
Destructive
No
Open world
Yes