I have a node js script that uses a variety of npm packages, the main one being puppeteer. This script navigates through a website (logs in to user account and books an appointment based on parameters I give it from a mongodb database). I want to automate this nodejs script to run at a certain time every day. Is Pipedream able to do this? My nodejs script has several functions and hundreds of lines of code.
Hi @kwachter,
First off, welcome to the Pipedream community. Happy to have you!
Yes I personally use Pipedream for scraping data from APIs and inserting them into my MongoDB instance for analysis.
One caveat with puppeteer specifically is that it’s not 100% compatible with our infrastructure. You can follow along with the developments on this issue and workarounds other Pipedreamers have found here:
Another alternative is to use a Python step in a v2 workflow to use BeautifulSoup to parse scraped HTML.
I hope this helps!
I have recorded a short video on how Pipedream can be used to run headless Chrome API . ScrapeNinja API, which is used in the video, is not exactly a generic Puppeteer instance, it has two API endpoints, /scrape and /scrape-js, where /scrape-js is essentially a “dumbed down” version of headless Chrome, which is much more convenient to use, as it is targeted at specific use case of web scraping websites. If your use case is not web scraping related, ScrapeNinja is probably not for you and the video will not be relevant.
Building low code web scraper to extract Hackernews titles to Google Sheets, via Pipedream:
Nice tutorial @restyler !
I didn’t know ScrapeNinja was a thing, it’s very easy to get started - and no servers to manage which is awesome.
I’ve just added an official Pipedream integration for it:
You can just provide your RapidAPI access token, then use Python or Node.js to scaffold quick web scraping steps.