Matlab Connect Databricks Spark Agent Skill

MATLAB Interface for Databricks — Spark (Databricks Connect)

Read data from Databricks into MATLAB via Databricks Connect. This skill covers first-time setup, authentication, and using Spark DataFrames from desktop MATLAB.

When to Use

• First-time setup of the MATLAB Interface for Databricks package
• Configuring .databrickscfg and authentication (OauthU2M, OauthM2M, PAT)
• Reading large tables via Spark with server-side filtering before pulling data locally
• Creating a Spark session from desktop MATLAB (getDatabricksSession)
• Converting Spark DataFrames to MATLAB tables or writing MATLAB tables back to Spark

When NOT to Use

• SQL queries on small-to-medium datasets via JDBC/ODBC — use /matlab-connect-databricks-jdbc
• Deploying compiled MATLAB code to Databricks clusters (Job workflow)
• Databricks Files API operations
• REST API management (Clusters, Jobs, Unity Catalog admin)

Workflow

• Obtain the package — Ask the user if they already have the MATLAB Interface for Databricks downloaded locally. If not, direct them to download it from https://www.mathworks.com/solutions/partners/databricks.html
• Setup — Run setup from the package directory to configure settings, .databrickscfg, and install the Databricks Connect library via pip
• Verify — Create a Spark session and run spark.range(1) to confirm connectivity
• Read — Use spark.read().table() to get a DataFrame handle (data stays on cluster)
• Filter — Apply .filter(), .withColumn(), .select() server-side
• Collect — Convert to MATLAB table with T = table(DF) only after reducing data

Key Functions

| Function | Purpose |

|----------|---------|

| setup | Interactive first-time configuration: settings, .databrickscfg, and Databricks Connect library download |

| startup | Adds package paths to MATLAB (run once per session, required after setup) |

| matlab.databricks.setup.configureDBC() | Re-run Databricks Connect library download independently (supports alternativeRepo for internal Artifactory) |

| getDatabricksSession() | Creates a databricks.PySparkSession connected to Databricks (classic compute) |

| getDatabricksSession(serverless=true) | Creates a serverless session (no cluster needed, 10-min timeout) |

| updateClusterId(id) | Updates the cluster ID in .databrickscfg (pass "serverless" to switch to serverless mode) |

| spark.range(n) | Creates a DataFrame with n rows (useful for testing) |

| spark.read().table("catalog.schema.table") | Reads a Unity Catalog table as a DataFrame |

| DF.filter(expr) | Server-side row filtering |

| DF.withColumn(name, col) | Adds/transforms a column server-side (col must be a matlab.pyspark.sql.column.Column) |

| DF.limit(n) | Server-side row limiting (returns first n rows as a new DataFrame) |

| matlab.pyspark.sql.functions.col(name) | Creates a Column reference by name |

| matlab.pyspark.sql.functions.lit(value) | Creates a Column containing a literal constant |

| DF.select(cols) | Server-side column selection |

| DF.show(n) | Displays n rows (stays on cluster) |

| DF.printSchema | Prints DataFrame schema |

| table(DF) | Converts DataFrame to a MATLAB table (pulls data locally) |

| spark.sql(query) | Executes a SQL query and returns a DataFrame |

| spark.table("catalog.schema.table") | Shortcut for spark.read().table() |

| matlab.sparkutils.table2dataset(T, spark) | Converts a MATLAB table back to a Spark DataFrame |

| DF.write.mode(m).format(f).save(path) | Writes DataFrame to a file path |

| DF.write.mode(m).format(f).saveAsTable(name) | Writes DataFrame to a Unity Catalog table |

Patterns

First-Time Setup

Run setup from the package's Software/MATLAB directory. It is interactive — follow the prompts to configure the host URL, authentication method, cluster ID, and Databricks Connect library installation.

The Databricks Connect library is not shipped with the package — setup downloads it via pip (from PyPI or an internal Artifactory) into a Python virtual environment under Software/MATLAB/Connect/<version>/venv/. This requires Python 3.10–3.12 and pip to be available.

If the Databricks Connect library download fails repeatedly, do not retry with modified arguments — ask the user to check with their IT team whether the required version of databricks-connect is available in their organization's Python package repository.

cd('/path/to/databricks-package/Software/MATLAB')
setup

After setup completes, MATLAB's pyenv must point to the venv Python — not the system Python. If getDatabricksSession fails with "databricks.connect package is not installed", switch pyenv:

terminate(pyenv);
pyenv(Version="/path/to/databricks-package/Software/MATLAB/Connect/17.3/venv/bin/python");

Warning: If Python has already been loaded in-process (ExecutionMode is "InProcess"), terminate(pyenv) will fail. A full MATLAB restart is required to switch pyenv in this case. Do not attempt to switch pyenv mid-session after Python has been used — it will not work and may disconnect the MCP server. Instead, ask the user to restart MATLAB, then set pyenv to the venv path before calling any Python-dependent functions.

Then run startup to add the package paths to MATLAB. This must also be run at the beginning of each new MATLAB session:

startup

Then verify connectivity:

spark = getDatabricksSession();
T = table(spark.range(1));
disp(T)

If this returns a 1-row table with column id = 0, authentication and compute are working.

Authentication Configuration

Authentication uses the .databrickscfg file — an INI-format configuration file located in the user's home directory (~/.databrickscfg). It stores the Databricks workspace URL, authentication credentials, and cluster ID. The file supports multiple named profiles so you can switch between workspaces or auth methods.

OauthU2M (recommended for interactive use) — requires only host. Note: OauthU2M opens a browser for login and requires pasting a redirect URL back into MATLAB. It does not work in non-interactive contexts (scripts, batch mode, MCP tools). Use OauthM2M or PAT for automation:

[DEFAULT]
host = https://<workspace-url>
cluster_id = <cluster-id>

OauthM2M (for service principals / automation) — requires host, client_id, client_secret:

[M2M]
host = https://<workspace-url>
client_id = <service-principal-client-id>
client_secret = <service-principal-secret>
cluster_id = <cluster-id>

PAT (simple but may be disabled by admins) — requires host and token:

[PAT]
host = https://<workspace-url>
token = <personal-access-token>
cluster_id = <cluster-id>

Specifying the auth method: When authMethod is not passed, getDatabricksSession uses a chain that tries methods in order until one succeeds — this may not pick what the user expects (e.g., it may trigger an OauthU2M browser prompt even if a PAT token is configured). Always ask the user which auth method they want and pass it explicitly:

spark = getDatabricksSession(authMethod="PAT");
spark = getDatabricksSession(authMethod="OauthU2M");
spark = getDatabricksSession(authMethod="OauthM2M");

To use a named profile other than [DEFAULT]:

spark = getDatabricksSession(profileName="M2M", authMethod="OauthM2M");

Environment variables override config file values:

| Variable | Overrides |

|----------|-----------|

| DATABRICKS_HOST | host |

| DATABRICKS_TOKEN | token |

| DATABRICKS_CLUSTER_ID | cluster_id |

| DATABRICKS_CLIENT_ID | client_id |

| DATABRICKS_CLIENT_SECRET | client_secret |

Create a Spark Session (Classic Compute)

spark = getDatabricksSession();

Uses the default profile's cluster_id. The cluster starts automatically if stopped (takes 3-5 minutes for a cold start).

To target a specific cluster:

spark = getDatabricksSession(cluster="0812-091301-zntwkr4b");

Create a Spark Session (Serverless Compute)

Serverless sessions don't require a cluster_id — they start instantly with a 10-minute inactivity timeout:

spark = getDatabricksSession(serverless=true);

To make serverless the default for a profile, remove the cluster_id and set serverless_compute_id:

updateClusterId("serverless");

Or configure .databrickscfg directly:

[DEFAULT]
host = https://<workspace-url>
serverless_compute_id = auto

Serverless requires Python 3.11–3.12 and Databricks Connect client >= 15.4.

Read and Filter a Table

Data stays on the cluster until explicitly collected. Filter server-side first, then collect.

spark = getDatabricksSession();

DF = spark.read().table("catalog.schema.sensor_readings");

filtered = DF.filter("temperature > 100 AND event_date > '2024-01-01'");

filtered.printSchema;
filtered.show(5);

T = table(filtered);

Filter and Limit Rows

Use .filter() for conditional filtering and .limit() for taking the first N rows:

spark = getDatabricksSession();
DF = spark.read().table("catalog.schema.large_table");

filtered = DF.filter("status = 'active' AND created_date > '2024-01-01'");
limited = filtered.limit(1000);
T = table(limited);

Add Computed Columns with withColumn

withColumn requires a matlab.pyspark.sql.column.Column object — not a scalar or string. Use col() to reference existing columns and lit() for constants:

import matlab.pyspark.sql.functions.col
import matlab.pyspark.sql.functions.lit

DF = spark.read().table("catalog.schema.measurements");

DF2 = DF.withColumn("temp_fahrenheit", col("temp_celsius") * lit(9/5) + lit(32));
DF3 = DF2.withColumn("source", lit("sensor_array"));

Do not pass raw MATLAB scalars or strings to withColumn — it will error. Always wrap values in lit().

Read Files from Unity Catalog Volumes

Use spark.read.format().load() for CSV, Parquet, or JSON files stored on Volumes:

spark = getDatabricksSession();

% CSV with header
DF = spark.read.format("csv").option("header", "true").load("/Volumes/catalog/schema/volume/data.csv");

% Parquet
DF = spark.read.format("parquet").load("/Volumes/catalog/schema/volume/data.parquet");

% JSON
DF = spark.read.format("json").load("/Volumes/catalog/schema/volume/data.json");

After loading, use the same filter/select/collect workflow as with tables.

Convert MATLAB Table Back to Spark DataFrame

DF_new = matlab.sparkutils.table2dataset(T, spark);

Write to a Volumes path as Parquet:

DF_new.write.mode("overwrite").format("parquet").save("/Volumes/catalog/schema/volume/output_data");

Write to a Unity Catalog table:

DF_new.write.mode("overwrite").format("delta").saveAsTable("catalog.schema.output_table");

Always specify .mode() — without it, writes fail if the target already exists. Options: "overwrite", "append", "ignore", "error" (default).

Handle Stale Sessions

If a session becomes unresponsive after inactivity:

clear spark
spark = getDatabricksSession(forceNewSession=true);

Python Version Compatibility

The local Python version must match the Databricks runtime's Python version:

| Runtime | Python Version | Notes |

|---------|---------------|-------|

| 18.x | 3.12 | Supports serverless and classic |

| 17.3 | 3.12 | Supports serverless and classic |

| 16.4 LTS | 3.12 | Supports serverless (>= 16.4.1) and classic |

| 15.4 LTS | 3.11 | Supports serverless (>= 15.4.10) and classic |

| 14.3 LTS | 3.10 | Classic only |

| 13.3 LTS | 3.10 | Classic only |

Check with:

pe = pyenv;
disp(pe.Version)

Conventions

• Always filter DataFrames server-side before calling table(DF) — pulling millions of unfiltered rows wastes bandwidth and memory
• Use three-level names for Unity Catalog tables: "catalog.schema.table"
• Use getDatabricksSession() — never construct Spark sessions manually or mimic PySpark builder patterns
• Use forceNewSession=true when a session becomes stale, not clear all
• Never hardcode tokens or secrets in MATLAB code — use .databrickscfg or environment variables
• Only Databricks Connect v2 is supported (v1 is not supported)

Common Mistakes

| Mistake | Why It's Wrong | Correct Approach |

|---------|---------------|-----------------|

| Manually building JDBC URLs with Simba driver params | Bypasses the package entirely; fragile and insecure | Use getDatabricksSession() for Spark or databricks.JDBCConnection for SQL |

| Inventing PySpark-like builder: databricks.spark.Session.builder().remote() | These classes do not exist in MATLAB | Use getDatabricksSession() which returns databricks.PySparkSession |

| Hardcoding tokens in source code | Security risk; tokens expire and rotate | Store credentials in .databrickscfg or environment variables |

| Calling table(DF) on unfiltered large tables | Transfers entire dataset to local MATLAB memory | Apply .filter() / .select() / .limit() first, then collect |

| Passing scalars to withColumn: DF.withColumn("x", 2) | Second argument must be a Column object | Use DF.withColumn("x", lit(2)) or DF.withColumn("x", col("y") * lit(2)) |

| Writing without .mode(): DF.write.save(path) | Fails if target exists (default mode is "error") | Use .mode("overwrite") or .mode("append") before .save() or .saveAsTable() |

| Using databricks.SCIM.me to verify auth | Deprecated API | Verify with spark = getDatabricksSession(); table(spark.range(1)) |

| Calling getDatabricksSession() without authMethod when user has PAT configured | Default chain tries methods in order and may trigger OauthU2M browser prompt instead of using PAT | Ask the user which auth method they want and pass authMethod="PAT" explicitly |

| Passing a DataFrame as an argument to a chained method: DF.unionAll(DF2) | The MATLAB Spark wrapper's bracket method only supports dot-chaining, not methods that take DataFrame arguments | Use spark.sql() with a SQL UNION ALL query instead, or if the data is small enough, collect both DataFrames to MATLAB tables, manipulate locally, and push back with table2dataset |

| Mismatched Python version | Session creation fails with version error | Match local Python to runtime version (e.g., 3.12 for runtime 17.3) |

| PAT token missing scopes | 403 "does not have required scopes: clusters" | Regenerate the PAT with All access or at minimum the clusters scope enabled |

| cluster_id not set error | .databrickscfg has no cluster_id for the active profile | Run updateClusterId("your-cluster-id") or add cluster_id to the profile manually |

----

Copyright 2026 The MathWorks, Inc.

----