databricks-deploy-integration

•

1.9k2593 months ago

Deploy Databricks jobs and pipelines with Asset Bundles. Use when deploying jobs to different environments, managing deployments, or setting up deployment automation. Trigger with phrases like "databricks deploy", "asset bundles", "databricks deployment", "deploy to production", "bundle deploy".

Quick Install

npxskills add jeremylongshore/claude-code-plugins-plus-skills--skill databricks-deploy-integration

Instructions

[!WARNING] DEPRECATED — to be removed in [email protected].

This v1 skill is replaced in the v2 rebuild. Migrate to: databricks-bundle-medic. See the pack README → Migration: v1 → v2 for the full map and rationale.

Databricks Deploy Integration

Overview

Deploy Databricks jobs, DLT pipelines, and ML models using Declarative Automation Bundles (DABs, formerly Asset Bundles). Bundles provide infrastructure-as-code with databricks.yml defining resources, targets (dev/staging/prod), variables, and permissions. The CLI handles validation, deployment, and lifecycle management.

Prerequisites

Databricks CLI v0.200+ (databricks --version)
Workspace access with service principal for automated deploys
databricks.yml bundle configuration at project root

Instructions

Step 1: Initialize a Bundle

Loading…

Related Skills

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,360

skill-creator

Create or update AgentSkills. Use when designing, structuring, or packaging skills with scripts, references, and assets.

367,309

1password

Set up and use 1Password CLI (op). Use when installing the CLI, enabling desktop app integration, signing in (single or multi-account), or reading/injecting/running secrets via op.

365,915

himalaya

CLI to manage emails via IMAP/SMTP. Use `himalaya` to list, read, write, reply, forward, search, and organize emails from the terminal. Supports multiple accounts and message composition with MML (MIME Meta Language).

365,885

Tags & Topics

aiautomationclaude-codedevopsmcpai-agents

# Create from a template
databricks bundle init

# Available templates:
# - default-python: Python notebook project
# - default-sql: SQL project
# - mlops-stacks: Full MLOps template with feature engineering

# databricks.yml — single source of truth for project deployment
bundle:
  name: sales-etl-pipeline

workspace:
  host: ${DATABRICKS_HOST}

variables:
  catalog:
    description: Unity Catalog name
    default: dev_catalog
  alert_email:
    description: Alert notification email
    default: [email protected]
  warehouse_size:
    default: "2X-Small"

include:
  - resources/*.yml

targets:
  dev:
    default: true
    mode: development
    # dev mode auto-prefixes resources with [username] and enables debug
    workspace:
      root_path: /Users/${workspace.current_user.userName}/.bundle/${bundle.name}/dev
    variables:
      catalog: dev_catalog

  staging:
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/staging
    variables:
      catalog: staging_catalog
      alert_email: [email protected]

  prod:
    mode: production
    # production mode prevents accidental destruction
    workspace:
      root_path: /Shared/.bundle/${bundle.name}/prod
    variables:
      catalog: prod_catalog
      alert_email: [email protected]
      warehouse_size: "Medium"

# resources/jobs.yml
resources:
  jobs:
    daily_etl:
      name: "daily-etl-${bundle.target}"
      max_concurrent_runs: 1
      timeout_seconds: 14400

      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "UTC"

      email_notifications:
        on_failure: ["${var.alert_email}"]

      tasks:
        - task_key: extract
          notebook_task:
            notebook_path: ./src/extract.py
            base_parameters:
              catalog: "${var.catalog}"
          job_cluster_key: etl

        - task_key: transform
          depends_on: [{task_key: extract}]
          notebook_task:
            notebook_path: ./src/transform.py
          job_cluster_key: etl

        - task_key: load
          depends_on: [{task_key: transform}]
          notebook_task:
            notebook_path: ./src/load.py
          job_cluster_key: etl

      job_clusters:
        - job_cluster_key: etl
          new_cluster:
            spark_version: "14.3.x-scala2.12"
            node_type_id: "i3.xlarge"
            autoscale:
              min_workers: 1
              max_workers: 4
            aws_attributes:
              availability: SPOT_WITH_FALLBACK
              first_on_demand: 1

# resources/pipelines.yml (DLT)
resources:
  pipelines:
    dlt_pipeline:
      name: "dlt-pipeline-${bundle.target}"
      target: "${var.catalog}.silver"
      catalog: "${var.catalog}"
      libraries:
        - notebook:
            path: ./src/dlt_pipeline.py
      continuous: false
      development: ${bundle.target == "dev"}

# Validate — checks YAML syntax, variable resolution, permissions
databricks bundle validate -t staging

# Deploy — creates/updates jobs, uploads notebooks, syncs config
databricks bundle deploy -t staging

# Summary — show what's deployed
databricks bundle summary -t staging

# Run — trigger a specific job/pipeline
databricks bundle run daily_etl -t staging

# Run and wait for completion
databricks bundle run daily_etl -t staging --restart-all-workflows

# Sync — live-reload files during development
databricks bundle sync -t dev --watch

# Destroy — remove all deployed resources (dev only!)
databricks bundle destroy -t dev --auto-approve

# 1. Validate staging is clean
databricks bundle validate -t staging

# 2. Deploy and test on staging
databricks bundle deploy -t staging
RUN=$(databricks bundle run daily_etl -t staging --output json | jq -r '.run_id')
databricks runs get --run-id $RUN | jq '.state.result_state'

# 3. After staging passes, deploy to production
databricks bundle validate -t prod
databricks bundle deploy -t prod

# 4. Verify production deployment
databricks bundle summary -t prod
databricks jobs list --output json | \
  jq '.[] | select(.settings.name | contains("daily-etl-prod"))'

# resources/jobs.yml — add permissions block
resources:
  jobs:
    daily_etl:
      name: "daily-etl-${bundle.target}"
      permissions:
        - group_name: data-engineers
          level: CAN_MANAGE
        - group_name: data-analysts
          level: CAN_VIEW
        - service_principal_name: cicd-service-principal
          level: CAN_MANAGE_RUN

Issue	Cause	Solution
`bundle validate` fails	Invalid YAML or unresolved variable	Check variable definitions and target config
`PERMISSION_DENIED` on deploy	Service principal lacks workspace access	Add SP to workspace in Account Console
`RESOURCE_CONFLICT`	Resource name collision across targets	Bundle auto-prefixes in `development` mode
`Cluster quota exceeded`	Too many active clusters	Use instance pools or terminate idle clusters
`Cannot destroy production`	`mode: production` prevents accidental destroy	This is intentional — remove mode or use `--force`

# Override a variable at deploy time
databricks bundle deploy -t prod --var="warehouse_size=Large"

databricks bundle destroy -t dev --auto-approve
databricks bundle deploy -t dev

databricks-deploy-integration

Databricks Deploy Integration

Overview

Prerequisites

Instructions

Step 1: Initialize a Bundle

Related Skills

skill-creator

skill-creator

1password

himalaya

Databricks Deploy Integration

Overview

Prerequisites

Instructions

Step 1: Initialize a Bundle

Step 2: Configure databricks.yml

Step 3: Define Resources

Step 4: Deploy Lifecycle Commands

Step 5: Promote Staging to Production

Step 6: Permissions in Bundles

Output

Error Handling

Examples

Override Variables per Target

Clean Slate Redeploy (Dev Only)

Resources

Next Steps

Related Skills

skill-creator

skill-creator

1password

himalaya

Related Skills

skill-creator

skill-creator

1password

himalaya

Step 2: Configure `databricks.yml`