with DigitalOcean Spaces and WebScraper.IO?
Emit new event when a file is deleted from a DigitalOcean Spaces bucket
Emit new event when a file is uploaded to a DigitalOcean Spaces bucket
Emit new event when a page scraping job has completed. See the docs here
Creates a scraping job (scrapes a sitemap). See the docs here
Creates a sitemap for the selected website. See the docs here
Retrieves a list of scraping jobs for a sitemap. See the docs here
DigitalOcean Spaces API permits you to manage object storage, allowing for the storage and serving of massive amounts of data. This API is great for backing up, archiving, and providing public access to data or assets. On Pipedream, you can use this API to automate file operations like uploads, downloads, and deletions, as well as manage permissions and metadata. You can integrate it with other services for end-to-end workflow automation.
import { S3 } from "@aws-sdk/client-s3";
import { ListBucketsCommand } from "@aws-sdk/client-s3";
export default defineComponent({
props: {
digitalocean_spaces: {
type: "app",
app: "digitalocean_spaces"
}
},
async run({ steps, $ }) {
console.log(this.digitalocean_spaces.$auth)
const s3Client = new S3({
forcePathStyle: false, // Configures to use subdomain/virtual calling format.
endpoint: `https://${this.digitalocean_spaces.$auth.region}.digitaloceanspaces.com`,
region: "us-east-1",
credentials: {
accessKeyId: this.digitalocean_spaces.$auth.key,
secretAccessKey: this.digitalocean_spaces.$auth.secret
}
});
const data = await s3Client.send(new ListBucketsCommand({}));
return data.Buckets;
},
})
The WebScraper.IO API allows you to programmatically perform web scraping tasks, extracting structured data from websites. With the API, you can automate the gathering of web content for analysis, monitoring, and integration with other data sources. In Pipedream, you can leverage this API to build workflows that process, analyze, and act on the data you scrape without writing code for backend infrastructure.
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
webscraper_io: {
type: "app",
app: "webscraper_io",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://api.webscraper.io/api/v1/sitemaps`,
params: {
api_token: `${this.webscraper_io.$auth.api_key}`,
},
})
},
})