Databricks Vector Search

databricks/databricks-agent-databricks-vector-search

Databricks Vector Search endpoints and indexes for RAG and semantic search; covers index types, search modes, end-to-end RAG patterns

This is a copy. The original lives at databricks/databricks-vector-search.

14k tokens

context cost

the whole folder, loaded on every use

files

instructions only

copies elsewhere

how many repositories repackaged it

236

stars on the repo

on the repository, not the skill itself

Install

one command, takes just this skill from the repository

npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-vector-search

The instruction itself

26 sections, as written by the author

Databricks Vector Search

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Patterns for creating, managing, and querying vector search indexes for RAG and semantic search applications.

When to Use

Use this skill when:

Building RAG (Retrieval-Augmented Generation) applications
Implementing semantic search or similarity matching
Creating vector indexes from Delta tables
Choosing between storage-optimized and standard endpoints
Querying vector indexes with filters

Overview

Databricks Vector Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.

| Component | Description |

|-----------|-------------|

| Endpoint | Compute resource hosting indexes (Standard or Storage-Optimized) |

| Index | Vector data structure for similarity search |

| Delta Sync | Auto-syncs with source Delta table |

| Direct Access | Manual CRUD operations on vectors |

Endpoint Types

|------|---------|----------|------|----------|

Index Types

|------|------------|------|----------|

Quick Start

Create Endpoint

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Create a standard endpoint
endpoint = w.vector_search_endpoints.create_endpoint(
    name="my-vs-endpoint",
    endpoint_type="STANDARD"  # or "STORAGE_OPTIMIZED"
)
# Note: Endpoint creation is asynchronous; check status with get_endpoint()

Create Delta Sync Index (Managed Embeddings)

# Source table must have: primary key column + text column
index = w.vector_search_indexes.create_index(
    name="catalog.schema.my_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.documents",
        "embedding_source_columns": [
            {
                "name": "content",  # Text column to embed
                "embedding_model_endpoint_name": "databricks-gte-large-en"
            }
        ],
        "pipeline_type": "TRIGGERED"  # or "CONTINUOUS"
    }
)

Query Index

results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content", "metadata"],
    query_text="What is machine learning?",
    num_results=5
)

for doc in results.result.data_array:
    score = doc[-1]  # Similarity score is last column
    print(f"Score: {score}, Content: {doc[1][:100]}...")

Common Patterns

Create Storage-Optimized Endpoint

# For large-scale, cost-effective deployments
endpoint = w.vector_search_endpoints.create_endpoint(
    name="my-storage-endpoint",
    endpoint_type="STORAGE_OPTIMIZED"
)

Delta Sync with Self-Managed Embeddings

# Source table must have: primary key + embedding vector column
index = w.vector_search_indexes.create_index(
    name="catalog.schema.my_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.documents",
        "embedding_vector_columns": [
            {
                "name": "embedding",  # Pre-computed embedding column
                "embedding_dimension": 768
            }
        ],
        "pipeline_type": "TRIGGERED"
    }
)

Direct Access Index

import json

# Create index for manual CRUD
index = w.vector_search_indexes.create_index(
    name="catalog.schema.direct_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DIRECT_ACCESS",
    direct_access_index_spec={
        "embedding_vector_columns": [
            {"name": "embedding", "embedding_dimension": 768}
        ],
        "schema_json": json.dumps({
            "id": "string",
            "text": "string",
            "embedding": "array<float>",
            "metadata": "string"
        })
    }
)

# Upsert data
w.vector_search_indexes.upsert_data_vector_index(
    index_name="catalog.schema.direct_index",
    inputs_json=json.dumps([
        {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"},
        {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"},
    ])
)

# Delete data
w.vector_search_indexes.delete_data_vector_index(
    index_name="catalog.schema.direct_index",
    primary_keys=["1", "2"]
)

Query with Embedding Vector

# When you have pre-computed query embedding
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "text"],
    query_vector=[0.1, 0.2, 0.3, ...],  # Your 768-dim vector
    num_results=10
)

Hybrid Search (Semantic + Keyword)

Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See references/search-modes.md for detailed guidance on choosing between ANN and hybrid search.

# Combines vector similarity with keyword matching
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_text="SPARK-12345 executor memory error",
    query_type="HYBRID",
    num_results=10
)

Filtering

Standard Endpoint Filters (Dictionary)

# filters_json uses dictionary format
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_text="machine learning",
    num_results=10,
    filters_json='{"category": "ai", "status": ["active", "pending"]}'
)

Storage-Optimized Filters (SQL-like)

Storage-Optimized endpoints use SQL-like filter syntax via the databricks-vectorsearch package's filters parameter (accepts a string):

from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()
index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")

# SQL-like filter syntax for storage-optimized endpoints
results = index.similarity_search(
    query_text="machine learning",
    columns=["id", "content"],
    num_results=10,
    filters="category = 'ai' AND status IN ('active', 'pending')"
)

# More filter examples
# filters="price > 100 AND price < 500"
# filters="department LIKE 'eng%'"
# filters="created_at >= '2024-01-01'"

Trigger Index Sync

# For TRIGGERED pipeline type, manually sync
w.vector_search_indexes.sync_index(
    index_name="catalog.schema.my_index"
)

Scan All Index Entries

# Retrieve all vectors (for debugging/export)
scan_result = w.vector_search_indexes.scan_index(
    index_name="catalog.schema.my_index",
    num_results=100
)

Reference Files

| Topic | File | Description |

|-------|------|-------------|

| Index Types | references/index-types.md | Detailed comparison of Delta Sync (managed/self-managed) vs Direct Access |

| End-to-End RAG | references/end-to-end-rag.md | Complete walkthrough: source table → endpoint → index → query → agent integration |

| Search Modes | references/search-modes.md | When to use semantic (ANN) vs hybrid search, decision guide |

| Operations | references/troubleshooting-and-operations.md | Monitoring, cost optimization, capacity planning, migration |

CLI Quick Reference

# List endpoints
databricks vector-search-endpoints list-endpoints

# Create endpoint (positional args: NAME ENDPOINT_TYPE)
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD

# List indexes on endpoint (positional arg: ENDPOINT_NAME)
databricks vector-search-indexes list-indexes my-endpoint

# Get index status (positional arg: INDEX_NAME)
databricks vector-search-indexes get-index catalog.schema.my_index

# Sync index (positional arg: INDEX_NAME)
databricks vector-search-indexes sync-index catalog.schema.my_index

# Delete index (positional arg: INDEX_NAME)
databricks vector-search-indexes delete-index catalog.schema.my_index

Common Issues

| Issue | Solution |

|-------|----------|

| Index sync slow | Use Storage-Optimized endpoints (20x faster indexing) |

| Query latency high | Use Standard endpoint for <100ms latency |

| filters_json not working | Storage-Optimized uses SQL-like string filters via databricks-vectorsearch package's filters parameter |

| Embedding dimension mismatch | Ensure query and index dimensions match |

| Index not updating | Check pipeline_type; use sync_index() for TRIGGERED |

| Out of capacity | Upgrade to Storage-Optimized (1B+ vectors) |

| query_vector truncated | Large vectors (e.g. 1024-dim) can be truncated when serialized as JSON. Use query_text instead (for managed embedding indexes), or use the Databricks SDK to pass raw vectors |

Embedding Models

Databricks provides built-in embedding models:

|-------|------------|----------------|----------|

# Use with managed embeddings
embedding_source_columns=[
    {
        "name": "content",
        "embedding_model_endpoint_name": "databricks-gte-large-en"
    }
]

Notes

Storage-Optimized is newer — better for most use cases unless you need <100ms latency
Delta Sync recommended — easier than Direct Access for most scenarios
Hybrid search — available for both Delta Sync and Direct Access indexes
columns_to_sync matters — only synced columns are available in query results; include all columns you need
Filter syntax differs by endpoint — Standard uses dict-format filters, Storage-Optimized uses SQL-like string filters. Use the databricks-vectorsearch package's filters parameter which accepts both formats
Management vs runtime — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use VectorSearchRetrieverTool

databricks-model-serving - Deploy agents that use VectorSearchRetrieverTool
databricks-agent-bricks - Knowledge Assistants use RAG over indexed documents
databricks-unstructured-pdf-generation - Generate documents to index in Vector Search
databricks-unity-catalog - Manage the catalogs and tables that back Delta Sync indexes
databricks-pipelines - Build Delta tables used as Vector Search sources

How to use it

Copy the folder

Take databricks/databricks-agent-databricks-vector-search from the repository into ~/.claude/skills for personal use, or into .claude/skills inside a project.

Check the name does not clash

The agent identifies a skill by the name field in its header. Two skills with the same name cannot sit side by side — one of them will be ignored.

Databricks Vector Search

Install

The instruction itself

Databricks Vector Search

When to Use

Overview

Endpoint Types

Index Types

Quick Start

Create Endpoint

Create Delta Sync Index (Managed Embeddings)

Query Index

Common Patterns

Create Storage-Optimized Endpoint

Delta Sync with Self-Managed Embeddings

Direct Access Index

Query with Embedding Vector

Hybrid Search (Semantic + Keyword)

Filtering

Standard Endpoint Filters (Dictionary)

Storage-Optimized Filters (SQL-like)

Trigger Index Sync

Scan All Index Entries

Reference Files

CLI Quick Reference

Common Issues

Embedding Models

Notes

Related Skills

How to use it