Create Vector Search Index with Databricks API

Pipedream makes it easy to connect APIs for Databricks and 2,800+ other apps remarkably fast.

Trigger workflow on

HTTP requests, schedules and app events

Next, do this

Create Vector Search Index with the Databricks API

No credit card required

▶

Watch us build a workflow

8 min

Watch now ➜

Trusted by 1,000,000+ developers from startups to Fortune 500 companies

Developers ♥ Pipedream

Getting Started#

Create a workflow to Create Vector Search Index with the Databricks API. When you configure and deploy the workflow, it will run on Pipedream's servers 24x7 for free.

Configure the Create Vector Search Index action
1. Connect your Databricks account
2. Configure Index Name
3. Select a Endpoint Name
4. Select a Index Type
5. Configure Primary Key
6. Optional- Configure Source Table
7. Optional- Configure Columns to Sync
8. Optional- Configure Embedding Source Columns
9. Optional- Configure Schema JSON
10. Optional- Select a Pipeline Type
Select a trigger to run your workflow on HTTP requests, schedules or app events
Deploy the workflow
Send a test event to validate your setup
Turn on the trigger

Integrations#

Create Vector Search Index with Databricks API on New Requests (Payload Only) from HTTP / Webhook API

HTTP / Webhook + Databricks

Try it

Create Vector Search Index with Databricks API on New Submission from Typeform API

Typeform + Databricks

Try it

Create Vector Search Index with Databricks API on New Submission (Instant) from Jotform API

Jotform + Databricks

Try it

Create Vector Search Index with Databricks API on New Scheduled Tasks from Pipedream API

Pipedream + Databricks

Try it

Create Vector Search Index with Databricks API on New Download Counts from npm API

npm + Databricks

Try it

Details#

This is a pre-built, source-available component from Pipedream's GitHub repo. The component is developed by Pipedream and the community, and verified and maintained by Pipedream.

To contribute an update to an existing component or create a new component, create a PR on GitHub. If you're new to Pipedream component development, you can start with quickstarts for trigger span and action development, and then review the component API reference.

Create Vector Search Index on Databricks

Description:Creates a new vector search index in Databricks. [See the documentation](https://docs.databricks.com/api/workspace/vectorsearchindexes/createindex)

Version:0.0.1

Key:databricks-create-vector-search-index

View on GitHub

Code#

import databricks from "../../databricks.app.mjs";
import utils from "../../common/utils.mjs";
import { ConfigurationError } from "@pipedream/platform";

export default {
  key: "databricks-create-vector-search-index",
  name: "Create Vector Search Index",
  description:
    "Creates a new vector search index in Databricks. [See the documentation](https://docs.databricks.com/api/workspace/vectorsearchindexes/createindex)",
  version: "0.0.1",
  type: "action",
  props: {
    databricks,
    name: {
      type: "string",
      label: "Index Name",
      description:
        "A unique name for the index (e.g., `main_catalog.docs.en_wiki_index`).",
    },
    endpointName: {
      propDefinition: [
        databricks,
        "endpointName",
      ],
    },
    indexType: {
      type: "string",
      label: "Index Type",
      description: "Type of index (`DELTA_SYNC` or `DIRECT_ACCESS`).",
      options: [
        "DELTA_SYNC",
        "DIRECT_ACCESS",
      ],
    },
    primaryKey: {
      type: "string",
      label: "Primary Key",
      description: "The primary key column for the index.",
    },
    sourceTable: {
      type: "string",
      label: "Source Table",
      description:
        "The Delta table backing the index (required for `DELTA_SYNC`).",
      optional: true,
    },
    columnsToSync: {
      type: "string[]",
      label: "Columns to Sync",
      description:
        "List of columns to sync from the source Delta table. Example: `[\"id\", \"text\"]` (required for `DELTA_SYNC`).",
      optional: true,
    },
    embeddingSourceColumns: {
      type: "string[]",
      label: "Embedding Source Columns",
      description:
        "List of embedding source column configs. Each entry is a JSON object string like `{ \"embedding_model_endpoint_name\": \"e5-small-v2\", \"name\": \"text\" }`.Provide when Databricks computes embeddings (DELTA_SYNC).",
      optional: true,
    },
    schemaJson: {
      type: "string",
      label: "Schema JSON",
      description:
        "The schema of the index in JSON format. Example: `{ \"columns\": [{ \"name\": \"id\", \"type\": \"string\" }, { \"name\": \"text_vector\", \"type\": \"array<double>\" }] }`. Required for `DIRECT_ACCESS` indexes.",
      optional: true,
    },
    pipelineType: {
      type: "string",
      label: "Pipeline Type",
      description: "Pipeline type for syncing (default: TRIGGERED).",
      options: [
        "TRIGGERED",
        "CONTINUOUS",
      ],
      optional: true,
      default: "TRIGGERED",
    },
  },

  async run({ $ }) {
    const payload = {
      name: this.name,
      endpoint_name: this.endpointName,
      index_type: this.indexType,
      primary_key: this.primaryKey,
    };

    if (this.indexType === "DELTA_SYNC") {
      if (this.schemaJson) {
        throw new ConfigurationError(
          "`Schema JSON` is not allowed when indexType is DELTA_SYNC.",
        );
      }
      if (!this.sourceTable) {
        throw new ConfigurationError(
          "sourceTable is required when indexType is DELTA_SYNC.",
        );
      }

      const columnsToSync = Array.isArray(this.columnsToSync)
        ? this.columnsToSync
        : utils.parseObject(this.columnsToSync);

      const embeddingSourceColumns = utils.parseObject(this.embeddingSourceColumns);
      const hasSource = Array.isArray(embeddingSourceColumns) && embeddingSourceColumns.length > 0;
      if (!hasSource) {
        throw new ConfigurationError(
          "embeddingSourceColumns is required when indexType is DELTA_SYNC.",
        );
      }

      const deltaSpec = {
        source_table: this.sourceTable,
        pipeline_type: this.pipelineType || "TRIGGERED",
      };
      if (Array.isArray(columnsToSync) && columnsToSync.length > 0) {
        deltaSpec.columns_to_sync = columnsToSync;
      }
      if (hasSource) {
        for (const [
          i,
          c,
        ] of embeddingSourceColumns.entries()) {
          if (!c?.name || !c?.embedding_model_endpoint_name) {
            throw new ConfigurationError(
              `embeddingSourceColumns[${i}] must include "name" and "embedding_model_endpoint_name"`,
            );
          }
        }
        deltaSpec.embedding_source_columns = embeddingSourceColumns;
      }
      payload.delta_sync_index_spec = deltaSpec;
    }

    else if (this.indexType === "DIRECT_ACCESS") {
      if (this.sourceTable || this.columnsToSync?.length || this.embeddingSourceColumns?.length) {
        throw new ConfigurationError(
          "`Source Table`,`Embedding Source Columns` and `Columns to Sync` are not allowed when indexType is DIRECT_ACCESS.",
        );
      }
      if (!this.schemaJson) {
        throw new ConfigurationError(
          "schemaJson is required when indexType is DIRECT_ACCESS.",
        );
      }
      payload.direct_access_index_spec = {
        schema_json: this.schemaJson,
      };
    }

    const response = await this.databricks.createVectorSearchIndex({
      data: payload,
      $,
    });

    $.export(
      "$summary",
      `Successfully created vector search index: ${response?.name || this.name}`,
    );
    return response;
  },
};

Configuration#

This component may be configured based on the props defined in the component code. Pipedream automatically prompts for input values in the UI and CLI.

Label	Prop	Type	Description
Databricks	`databricks`	`app`	This component uses the Databricks app.
Index Name	`name`	`string`	A unique name for the index (e.g., `main_catalog.docs.en_wiki_index`).
Endpoint Name	`endpointName`	`string`	Select a value from the drop down menu.
Index Type	`indexType`	`string`	Select a value from the drop down menu:`DELTA_SYNCDIRECT_ACCESS`
Primary Key	`primaryKey`	`string`	The primary key column for the index.
Source Table	`sourceTable`	`string`	The Delta table backing the index (required for `DELTA_SYNC`).
Columns to Sync	`columnsToSync`	`string[]`	List of columns to sync from the source Delta table. Example: `["id", "text"]` (required for `DELTA_SYNC`).
Embedding Source Columns	`embeddingSourceColumns`	`string[]`	List of embedding source column configs. Each entry is a JSON object string like `{ "embedding_model_endpoint_name": "e5-small-v2", "name": "text" }`.Provide when Databricks computes embeddings (DELTA_SYNC).
Schema JSON	`schemaJson`	`string`	The schema of the index in JSON format. Example: `{ "columns": [{ "name": "id", "type": "string" }, { "name": "text_vector", "type": "array<double>" }] }`. Required for `DIRECT_ACCESS` indexes.
Pipeline Type	`pipelineType`	`string`	Select a value from the drop down menu:`TRIGGEREDCONTINUOUS`