Extract Web Data on Scale
Uses the ScrapeNinja real Chrome browser engine to scrape pages that require JS rendering. See the documentation
Use high-performance web scraping endpoint with Chrome browser TLS fingerprint, but without JavaScript execution and real browser overhead. See the documentation
Gets details of the authenticated user. See the docs here
ScrapeNinja API on Pipedream allows you to craft powerful serverless workflows for web scraping without the hassle of managing proxies or browsers. It's a tool that can extract data from websites, handling JavaScript rendering and anti-bot measures with ease. By integrating ScrapeNinja with Pipedream, you can automate data collection, collate and process the scraped data, and connect it to numerous other services for further analysis, alerting, or storage.
import { axios } from '@pipedream/platform';
export default defineComponent({
props: {
scrapeninja: {
type: "app",
app: "scrapeninja",
}
},
async run({steps, $}) {
return await axios($, {
method: 'POST',
url: 'https://scrapeninja.p.rapidapi.com/scrape',
headers: {
'content-type': 'application/json',
'X-RapidAPI-Key': this.scrapeninja.$auth.rapid_api_key,
'X-RapidAPI-Host': 'scrapeninja.p.rapidapi.com'
},
data: {
url:"https://news.ycombinator.com/"
}
})
},
})
The SmugMug API provides programmatic access to a user's SmugMug account, allowing them to manage photos, albums, and account settings. With Pipedream, you can automate tasks such as uploading new images, synchronizing photo galleries with other platforms, or triggering actions based on account activity. The API's capabilities paired with Pipedream's serverless platform enable you to craft custom workflows that react to events in SmugMug or orchestrate tasks across multiple apps.
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
smugmug: {
type: "app",
app: "smugmug",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.smugmug.com/api/v2!authuser`,
headers: {
"Accept": `application/json`,
},
}, {
token: {
key: this.smugmug.$auth.oauth_access_token,
secret: this.smugmug.$auth.oauth_refresh_token,
},
oauthSignerUri: this.smugmug.$auth.oauth_signer_uri,
})
},
})