with Crawlbase and News API?
Search through millions of articles from over 150,000 large and small news sources and blogs. See the documentation
Retrieve live top and breaking headlines for a category, single source, multiple sources, or keywords. See the documentation
The Crawlbase API provides powerful tools for web scraping and data extraction from any webpage. It handles large-scale data collection tasks, bypassing bot protection and CAPTCHAs, and returning structured data. Within Pipedream, you can leverage Crawlbase to automate the harvesting of web data, integrate scraped content with other services, and process it for analysis, reporting, or triggering other workflows.
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
crawlbase: {
type: "app",
app: "crawlbase",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.crawlbase.com/account`,
params: {
token: `${this.crawlbase.$auth.api_token}`,
product: `crawling-api`,
},
})
},
})
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
news_api: {
type: "app",
app: "news_api",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://newsapi.org/v2/everything`,
headers: {
"X-Api-Key": `${this.news_api.$auth.api_key}`,
},
params: {
"q": `Computer Science`,
},
})
},
})