What do you want to automate
with WebScraping.AI and Puppeteer?
Prompt, edit and deploy AI agents that connect to WebScraping.AI, Puppeteer and 2,500+ other apps in seconds.
Trusted by 1,000,000+ developers from startups to Fortune 500 companies
Popular WebScraping.AI and Puppeteer Actions#
Gets an answer to a question about a given webpage. See the documentation
Get the HTML of a webpage using Puppeteer. See the documentation for details.
Returns the full HTML content of a webpage specified by the URL. See the documentation:
Get the title of a webpage using Puppeteer. See the documentation
Returns the visible text content of a webpage specified by the URL. See the documentation
Overview of WebScraping.AI#
WebScraping.AI API provides powerful tools for extracting data from websites, enabling users to retrieve structured information without the hassle of setting up a custom scraper. It handles proxy rotation, browsers, and CAPTCHAs, allowing you to focus on data collection. With Pipedream, you can harness this capability to create automated workflows that trigger on various events, process web content, and connect with countless other apps to feed data pipelines, monitor changes, or populate databases.
Connect WebScraping.AI#
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
webscraping_ai: {
type: "app",
app: "webscraping_ai",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.webscraping.ai/account`,
params: {
api_key: `${this.webscraping_ai.$auth.api_key}`,
},
})
},
})
Overview of Puppeteer#
Puppeteer is a Node.js library which provides a high-level API to control Chrome/Chromium over the DevTools Protocol. Puppeteer runs in headless mode on Chromium on Pipedream.
Using Puppeteer you can perform tasks including:
- Capture Screenshots: Convert webpages into images.
- Processing PDFs: parse and scan PDFs.
- Web Scraping: Extract data from websites.
- UI/UX Testing: Verify user interface and experience.
- Integration with Test Frameworks: Combine with testing frameworks.
- Task Automation: Automate web-related tasks like form filling.
- Functional Testing: Automate user interactions to test web application functionality.
- Regression Testing: Ensure new code changes don't introduce bugs.
Connect Puppeteer#
import { puppeteer } from '@pipedream/browsers';
export default defineComponent({
async run({steps, $}) {
const browser = await puppeteer.browser();
// Interact with the web page programmatically
// See Puppeeter's Page documentation for available methods:
// https://pptr.dev/api/puppeteer.page
const page = await browser.newPage();
await page.goto('https://pipedream.com/');
const title = await page.title();
const content = await page.content();
// The browser needs to be closed, otherwise the step will hang
await browser.close();
return { title, content }
},
})Trusted by 1,000,000+ developers from startups to Fortune 500 companies