CONNECT APP

Build with ScrapeGraphAI

API Scraper powered by AI

Data Analytics

API key

Visit ScrapeGraphAI ↗ Start building

MCP

Give your agent ScrapeGraphAI tools

Every ScrapeGraphAI action is exposed as an MCP tool on Pipedream's remote server. Point a client at it with your end user's ID and Connect resolves that user's ScrapeGraphAI account for each tool call — you store no tokens.

Pipedream Connect

// accessToken: mint a short-lived token with the Connect SDK — see the MCP guide
const transport = new StreamableHTTPClientTransport(
  new URL("https://remote.mcp.pipedream.net/v3"),
  {
    requestInit: {
      headers: {
        Authorization: `Bearer ${accessToken}`,
        "x-pd-project-id": "{project_id}",
        "x-pd-environment": "production",
        "x-pd-external-user-id": "{external_user_id}", // any stable ID for this user in your system
        "x-pd-app-slug": "scrapegraphai",
      },
    },
  },
)

const mcp = new Client({ name: "my-agent", version: "1.0.0" })
await mcp.connect(transport)

const { tools } = await mcp.listTools()

// e.g. run Start Local Scraper:
const result = await mcp.callTool({
  name: "scrapegraphai-start-local-scraper",
  arguments: {
    html: "HTML",
    prompt: "Prompt",
  },
})

# access_token: mint a short-lived token with the Connect SDK — see the MCP guide
headers = {
    "Authorization": f"Bearer {access_token}",
    "x-pd-project-id": "{project_id}",
    "x-pd-environment": "production",
    "x-pd-external-user-id": "{external_user_id}",  # any stable ID for this user in your system
    "x-pd-app-slug": "scrapegraphai",
}

async with streamablehttp_client("https://remote.mcp.pipedream.net/v3", headers=headers) as (read, write, _):
    async with ClientSession(read, write) as session:
        await session.initialize()
        tools = await session.list_tools()

        # e.g. run Start Local Scraper:
        result = await session.call_tool("scrapegraphai-start-local-scraper", {
            "html": "HTML",
            "prompt": "Prompt",
        })

MCP guide →

API PROXY

Call the ScrapeGraphAI API directly

For an endpoint with no pre-built tool, the Connect proxy forwards your request to the ScrapeGraphAI API with the connected user's credentials attached. You store no tokens and write no refresh logic.

Pipedream Connect

const resp = await pd.proxy.get({
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  accountId: "apn_xxxxxxx",
  url: "https://api.scrapegraphai.com/v1/credits",
})

// Any allowed ScrapeGraphAI endpoint works here. Pipedream attaches the
// connected account's credentials to the outgoing request.

# The path segment is the target URL, URL-safe base64 encoded:
# https://api.scrapegraphai.com/v1/credits

curl "https://api.pipedream.com/v1/connect/{project_id}/proxy/aHR0cHM6Ly9hcGkuc2NyYXBlZ3JhcGhhaS5jb20vdjEvY3JlZGl0cw?external_user_id={external_user_id}&account_id=apn_xxxxxxx" \
  -H "Authorization: Bearer {access_token}" \
  -H "x-pd-environment: production"

API proxy guide →

SDK

Run ScrapeGraphAI actions from your backend

Connect a user's ScrapeGraphAI account once, then run Start Local Scraper on their behalf from your own code — TypeScript, Python, or plain HTTP.

Pipedream Connect

import { PipedreamClient } from "@pipedream/sdk"

const pd = new PipedreamClient({
  projectId: process.env.PIPEDREAM_PROJECT_ID!,
  clientId: process.env.PIPEDREAM_CLIENT_ID!,
  clientSecret: process.env.PIPEDREAM_CLIENT_SECRET!,
  projectEnvironment: "production",
})

const result = await pd.actions.run({
  id: "scrapegraphai-start-local-scraper",
  externalUserId: "{external_user_id}", // any stable ID for this user in your system
  configuredProps: {
    scrapegraphai: { authProvisionId: "apn_xxxxxxx" },
    html: "HTML",
    prompt: "Prompt",
  },
})

from pipedream import Pipedream

pd = Pipedream(
    client_id="{oauth_client_id}",
    client_secret="{oauth_client_secret}",
    project_id="{project_id}",
    project_environment="production",
)

result = pd.actions.run(
    id="scrapegraphai-start-local-scraper",
    external_user_id="{external_user_id}",  # any stable ID for this user in your system
    configured_props={
        "scrapegraphai": {"authProvisionId": "apn_xxxxxxx"},
        "html": "HTML",
        "prompt": "Prompt",
    },
)

curl -X POST https://api.pipedream.com/v1/connect/{project_id}/actions/run \
  -H "Content-Type: application/json" \
  -H "X-PD-Environment: production" \
  -H "Authorization: Bearer {access_token}" \
  -d '{
    "external_user_id": "{external_user_id}",
    "id": "scrapegraphai-start-local-scraper",
    "configured_props": {
      "scrapegraphai": { "authProvisionId": "apn_xxxxxxx" },
      "html": "HTML",
      "prompt": "Prompt"
    }
  }'