Integrate the AWS API with the Databricks API

Setup the AWS API trigger to run a workflow which integrates with the Databricks API. Pipedream's integration platform allows you to integrate AWS and Databricks remarkably fast. Free for developers.

Start for free — no credit card required

Watch a demo

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Popular Ways to Connect AWS with Databricks#

Get Run Output with Databricks API on New Scheduled Tasks from AWS API

AWS + Databricks

Try it

Get Run Output with Databricks API on New SNS Messages from AWS API

AWS + Databricks

Try it

List Runs with Databricks API on New Scheduled Tasks from AWS API

AWS + Databricks

Try it

List Runs with Databricks API on New SNS Messages from AWS API

AWS + Databricks

Try it

Run Job Now with Databricks API on New Scheduled Tasks from AWS API

AWS + Databricks

Try it

Popular AWS and Databricks Triggers#

New Scheduled Tasks from the AWS API

Creates a Step Function State Machine to publish a message to an SNS topic at a specific timestamp. The SNS topic delivers the message to this Pipedream source, and the source emits it as a new event.

Try it

New SNS Messages from the AWS API

Creates an SNS topic in your AWS account. Messages published to this topic are emitted from the Pipedream source.

Try it

New Inbound SES Emails from the AWS API

The source subscribes to all emails delivered to a specific domain configured in AWS SES. When an email is sent to any address at the domain, this event source emits that email as a formatted event. These events can trigger a Pipedream workflow and can be consumed via SSE or REST API.

Try it

New Deleted S3 File from the AWS API

Emit new event when a file is deleted from a S3 bucket

Try it

New DynamoDB Stream Event from the AWS API

Emit new event when a DynamoDB stream receives new events. See the docs here

Try it

Popular AWS and Databricks Actions#

Get Run Output with the Databricks API

Retrieve the output and metadata of a single task run. See the documentation

Try it

List Runs with the Databricks API

Lists all runs available to the user. See the documentation

Try it

Run Job Now with the Databricks API

Run a job now and return the id of the triggered run. See the documentation

Try it

CloudWatch Logs - Put Log Event with the AWS API

Uploads a log event to the specified log stream. See docs

Try it

DynamoDB - Create Table with the AWS API

Creates a new table to your account. See docs

Try it

Overview of AWS#

The AWS API unlocks endless possibilities for automation with Pipedream. With this powerful combo, you can manage your AWS services and resources, automate deployment workflows, process data, and react to events across your AWS infrastructure. Pipedream offers a serverless platform for creating workflows triggered by various events that can execute AWS SDK functions, making it an efficient tool to integrate, automate, and orchestrate tasks across AWS services and other apps.

Connect AWS#

import AWS from 'aws-sdk'

export default defineComponent({
  props: {
    aws: {
      type: "app",
      app: "aws",
    }
  },
  async run({steps, $}) {
    const { accessKeyId, secretAccessKey } = this.aws.$auth
    
    /* Now, pass the accessKeyId and secretAccessKey to the constructor for your desired service. For example:
    
    const dynamodb = new AWS.DynamoDB({
      accessKeyId, 
      secretAccessKey,
      region: 'us-east-1',
    })
    
    */
  },
})

Overview of Databricks#

The Databricks API allows you to interact programmatically with Databricks services, enabling you to manage clusters, jobs, notebooks, and other resources within Databricks environments. Through Pipedream, you can leverage these APIs to create powerful automations and integrate with other apps for enhanced data processing, transformation, and analytics workflows. This unlocks possibilities like automating cluster management, dynamically running jobs based on external triggers, and orchestrating complex data pipelines with ease.

Connect Databricks#

import { axios } from "@pipedream/platform"
export default defineComponent({
  props: {
    databricks: {
      type: "app",
      app: "databricks",
    }
  },
  async run({steps, $}) {
    return await axios($, {
      url: `https://${this.databricks.$auth.domain}.cloud.databricks.com/api/2.0/clusters/list`,
      headers: {
        Authorization: `Bearer ${this.databricks.$auth.access_token}`,
      },
    })
  },
})

Start for free — no credit card required

Watch a demo