Databricks

Databricks is the lakehouse company, helping data teams solve the world’s toughest problems.

Integrate the Databricks API with the Filter API

Setup the Databricks API trigger to run a workflow which integrates with the Filter API. Pipedream's integration platform allows you to integrate Databricks and Filter remarkably fast. Free for developers.

Get Run Output with the Databricks API

Retrieve the output and metadata of a single task run. See the documentation

 
Try it
Continue execution if a condition Is met with the Filter API

Continue workflow execution only if a condition is met

 
Try it
List Runs with the Databricks API

Lists all runs available to the user. See the documentation

 
Try it
End execution if a condition is met with the Filter API

End workflow execution if a condition is met

 
Try it
Run Job Now with the Databricks API

Run a job now and return the id of the triggered run. See the documentation

 
Try it

Connect Databricks

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import { axios } from "@pipedream/platform"
export default defineComponent({
  props: {
    databricks: {
      type: "app",
      app: "databricks",
    }
  },
  async run({steps, $}) {
    return await axios($, {
      url: `https://${this.databricks.$auth.domain}.cloud.databricks.com/api/2.0/clusters/list`,
      headers: {
        Authorization: `Bearer ${this.databricks.$auth.access_token}`,
      },
    })
  },
})

Overview of Filter

The Filter API is a great way to build powerful workflows that can manipulate
and transform data. Here are some examples of what you can build using the
Filter API:

  • A workflow that transforms data from one format to another
  • A workflow that filters out data that does not meet certain criteria
  • A workflow that sorts data in a specific order
  • A workflow that calculates statistics or aggregates data
  • A workflow that combines data from multiple sources

Connect Filter

1
2
3
4
5
6
7
8
9
10
export default defineComponent({
  async run({ steps, $ }) {
    let condition = false
    if (condition == false) {
      $.flow.exit("Ending workflow early because the condition is false")
    } else {
        $.export("$summary", "Continuing workflow, since condition for ending was not met.")
    }
  },
})