Configure Databricks local development with dbx, Databricks Connect, and IDE.
Use when setting up a local dev environment, configuring test workflows,
or establishing a fast iteration cycle with Databricks.
Trigger with phrases like "databricks dev setup", "databricks local",
"databricks IDE", "develop with databricks", "databricks connect".
This v1 skill is replaced in the v2 rebuild. Migrate to:databricks-bundle-medic.
See the pack README → Migration: v1 → v2 for the full map and rationale.
Databricks Local Dev Loop
Overview
Set up a fast local development workflow using Databricks Connect v2, Asset Bundles, and VS Code. Databricks Connect lets you run PySpark code locally while executing on a remote Databricks cluster, giving you IDE debugging, fast iteration, and proper test isolation.
Prerequisites
Completed databricks-install-auth setup
Python 3.10+ (must match cluster's Python version)
set -euo pipefail
# Create virtual environment
python -m venv .venv && source .venv/bin/activate
# Databricks Connect v2 — version MUST match cluster DBR
pip install "databricks-connect==14.3.*"
# SDK and CLI
pip install databricks-sdk
# Testing
pip install pytest pytest-cov
# Verify Connect installation
databricks-connect test
Step 3: Configure Databricks Connect
Databricks Connect v2 reads from standard SDK auth (env vars, ~/.databrickscfg, or DATABRICKS_CLUSTER_ID).
# Set cluster for Connect to use
export DATABRICKS_HOST="https://adb-1234567890123456.7.azuredatabricks.net"
export DATABRICKS_TOKEN="dapi..."
export DATABRICKS_CLUSTER_ID="0123-456789-abcde123"
# src/utils/spark_session.py
from databricks.connect import DatabricksSession
def get_spark():
"""Get a DatabricksSession — runs Spark on the remote cluster."""
return DatabricksSession.builder.getOrCreate()
# Usage: df operations execute on the remote cluster
spark = get_spark()
df = spark.sql("SELECT current_timestamp() AS now")
df.show() # Results streamed back locally
# tests/conftest.py
import pytest
from pyspark.sql import SparkSession
@pytest.fixture(scope="session")
def local_spark():
"""Local SparkSession for fast unit tests (no cluster needed)."""
return (
SparkSession.builder
.master("local[*]")
.appName("unit-tests")
.config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
.config("spark.sql.catalog.spark_catalog",
"org.apache.spark.sql.delta.catalog.DeltaCatalog")
.getOrCreate()
)
@pytest.fixture(scope="session")
def remote_spark():
"""DatabricksSession for integration tests (requires running cluster)."""
from databricks.connect import DatabricksSession
return DatabricksSession.builder.getOrCreate()
# tests/unit/test_transforms.py
def test_dedup_by_primary_key(local_spark):
from src.pipelines.silver import dedup_by_key
data = [("a", 1), ("a", 2), ("b", 3)]
df = local_spark.createDataFrame(data, ["id", "value"])
result = dedup_by_key(df, key_col="id", order_col="value")
assert result.count() == 2
# Keeps latest value per key
assert result.filter("id = 'a'").first()["value"] == 2
Step 6: Dev Workflow Commands
# Validate bundle configuration
databricks bundle validate
# Deploy dev resources to workspace
databricks bundle deploy -t dev
# Run a job
databricks bundle run daily_etl -t dev
# Sync local files to workspace (live reload)
databricks bundle sync -t dev --watch
# Run local unit tests (fast, no cluster)
pytest tests/unit/ -v
# Run integration tests (needs cluster)
pytest tests/integration/ -v --tb=short
# Full test with coverage
pytest tests/ --cov=src --cov-report=html
