Databricks
Databricks is the lakehouse company, helping data teams solve the world’s toughest problems.
Integrate the Databricks API with the MongoDB API
Setup the Databricks API trigger to run a workflow which integrates with the MongoDB API. Pipedream's integration platform allows you to integrate Databricks and MongoDB remarkably fast. Free for developers.
Popular Databricks and MongoDB Actions#
Retrieve the output and metadata of a single task run. See the documentation
Create a new document in a collection of your choice. See the docs here
Run a job now and return the id of the triggered run. See the documentation
Overview of Databricks#
The Databricks API allows you to interact programmatically with Databricks services, enabling you to manage clusters, jobs, notebooks, and other resources within Databricks environments. Through Pipedream, you can leverage these APIs to create powerful automations and integrate with other apps for enhanced data processing, transformation, and analytics workflows. This unlocks possibilities like automating cluster management, dynamically running jobs based on external triggers, and orchestrating complex data pipelines with ease.
Connect Databricks#
import { axios } from "@pipedream/platform"
export default defineComponent({
props: {
databricks: {
type: "app",
app: "databricks",
}
},
async run({steps, $}) {
return await axios($, {
url: `https://${this.databricks.$auth.domain}.cloud.databricks.com/api/2.0/clusters/list`,
headers: {
Authorization: `Bearer ${this.databricks.$auth.access_token}`,
},
})
},
})
Overview of MongoDB#
The MongoDB API provides powerful capabilities to interact with a MongoDB database, allowing you to perform CRUD (Create, Read, Update, Delete) operations, manage databases, and execute sophisticated queries. With Pipedream, you can harness these abilities to automate tasks, sync data across various apps, and react to events in real-time. It’s a combo that’s particularly potent for managing data workflows, syncing application states, or triggering actions based on changes to your data.
Connect MongoDB#
import mongodb from 'mongodb'
export default defineComponent({
props: {
mongodb: {
type: "app",
app: "mongodb",
},
collection: {
type: "string"
},
filter: {
type: "object"
}
},
async run({steps, $}) {
const MongoClient = mongodb.MongoClient
const {
database,
hostname,
username,
password,
} = this.mongodb.$auth
const url = `mongodb+srv://${username}:${password}@${hostname}/test?retryWrites=true&w=majority`
const client = await MongoClient.connect(url, {
useNewUrlParser: true,
useUnifiedTopology: true
})
const db = client.db(database)
const results = await db.collection(this.collection).find(this.filter).toArray();
$.export('results', results);
await client.close()
},
})