with Oxylabs and FireCrawl?
Emit new event when a new scheduled run is completed. See the documentation
Establish a proxy session using the Residential Proxy endpoint. See the documentation
Crawls a given URL and returns the contents of sub-pages. See the documentation
Extract structured data from one or multiple URLs. See the documentation
Obtains the status and data from a previous crawl operation. See the documentation
The Oxylabs API provides robust tools for web data extraction, enabling you to gather vast amounts of data efficiently from various sources across the web. With Pipedream's serverless platform, you can harness this power to create automated workflows that trigger on specific events, process the data, and connect to other services. Whether you're monitoring brand sentiment, tracking prices, or conducting market research, integrating Oxylabs with Pipedream can significantly streamline your data collection and analysis tasks.
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
oxylabs: {
type: "app",
app: "oxylabs",
}
},
async run({steps, $}) {
const data = {
"geo_location": `United States`,
"url": `https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html`,
"source": `universal`,
}
return await axios($, {
method: "post",
url: `https://${this.oxylabs.$auth.api_name}.oxylabs.io/v1/queries`,
headers: {
"Content-Type": `application/json`,
},
auth: {
username: `${this.oxylabs.$auth.username}`,
password: `${this.oxylabs.$auth.password}`,
},
data,
})
},
})
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
firecrawl: {
type: "app",
app: "firecrawl",
}
},
async run({steps, $}) {
const data = {
"url": "https://pipedream.com",
}
return await axios($, {
method: "post",
url: `https://api.firecrawl.dev/v0/crawl`,
headers: {
Authorization: `Bearer ${this.firecrawl.$auth.api_key}`,
},
data,
})
},
})