The easiest way to scrape websites via #API. ScrapingAnt uses the latest Chrome browser and rotates proxies to automate your data mining tasks.
Send a request using the standard extraction method of ScrapingAnt. See the documentation
Convert an object to a JSON format string
Add or subtract time from a given input
Get the duration between two dates in days, hours, minutes, and seconds along with checking if they are the same.
The ScrapingAnt API allows you to scrape web pages without getting blocked. It can handle JavaScript rendering, cookies, sessions, and can even interact with web pages as if a real person were browsing. Using Pipedream, you can integrate ScrapingAnt with countless other apps to automate data extraction and feed this data into various business processes, analytics tools, or databases.
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
scrapingant: {
type: "app",
app: "scrapingant",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.scrapingant.com/v2/general`,
headers: {
"x-api-key": `${this.scrapingant.$auth.api_token}`,
},
params: {
url: `pipedream.com`,
},
})
},
})
The Pipedream Utils app is a set of pre-built functions that streamline common tasks in your workflows. It acts like a Swiss Army knife for developers, providing essential tools such as format conversion, date manipulation, and text processing. By leveraging these functions, you can reduce the boilerplate code needed for routine operations, speeding up the development of intricate automations. The Helper Functions API can be a game changer when it comes to tasks like parsing dates in user-friendly formats, encoding and decoding data, or generating UUIDs, making them more efficient and less error-prone.
export default defineComponent({
props: {
pipedream_utils: {
type: "app",
app: "pipedream_utils",
}
},
async run({steps, $}) {
},
})