with Scrapfly and Scrape-It.Cloud?
Automate content extraction from any text-based source using AI, LLM, and custom parsing. See the documentation
Initiate the scraping process for a specific endpoint. See the documentation here
Retrieve current subscription and account usage details from Scrapfly. See the documentation
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
scrapfly: {
type: "app",
app: "scrapfly",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.scrapfly.io/scrape`,
params: {
key: `${this.scrapfly.$auth.api_key}`,
url: `https://pipedream.com`,
render_js: `true`,
asp: `true`,
},
})
},
})
The Scrape-It.Cloud API allows you to automate the extraction of data from websites. It can parse, scrape, and retrieve content without the need for manual intervention. With this API on Pipedream, you can build workflows that trigger on various events and use the scraped data for numerous applications like data analysis, lead generation, and content aggregation.
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
scrape_it_cloud: {
type: "app",
app: "scrape_it_cloud",
}
},
async run({steps, $}) {
const data = {
"url": `https://pipedream.com`,
}
return await axios($, {
method: "post",
url: `https://api.scrape-it.cloud/scrape`,
headers: {
"Content-Type": `application/json`,
"x-api-key": `${this.scrape_it_cloud.$auth.api_key}`,
},
data,
})
},
})