Stack conventions

Databricks (PySpark, Delta Lake, Asset Bundles)

Databricks code: PySpark jobs and libraries, Lakeflow (Delta Live Tables) pipelines, notebooks kept as .py or .sql source, Databricks Asset Bundles (databricks.yml), Delta Lake tables and Unity Catalog objects; tested locally with pytest and a local SparkSession.

Databricks code: PySpark jobs and libraries, Lakeflow (Delta Live Tables) pipelines, notebooks kept as .py or .sql source, Databricks Asset Bundles (databricks.yml), Delta Lake tables and Unity Catalog objects; tested locally with pytest and a local SparkSession.

Used for: databricks.

Principles

• Put transformation logic in importable Python modules (src//) as functions from DataFrame to DataFrame; notebooks and job entry points only read parameters, call those functions and write results.
• Unit-test every transformation with pytest and a local SparkSession (master local[*] or local[1], spark.ui.enabled false, few shuffle partitions) and compare DataFrames with assertDataFrameEqual or chispa; tests never touch a workspace.
• Deploy with Databricks Asset Bundles (databricks.yml) per target (dev, staging, prod); jobs, pipelines and permissions are code reviewed in the pull request, never created by hand in the workspace.
• Address every table by its three-level Unity Catalog name (catalog.schema.table) with the catalog from a bundle variable or job parameter, so the same code runs in every environment.
• Never put a token or password in code, notebooks, bundle files or job parameters: read secrets with dbutils.secrets.get from a secret scope, and authenticate CI with OAuth (a service principal), not a personal access token.
• Write Delta tables with explicit schemas, MERGE for upserts with a deterministic match condition, and schema evolution only where it is intended.
• Keep work on the executors: no collect() or toPandas() on large data, no Python UDF where a built-in function exists, and broadcast only small dimensions.
• Grant least privilege in Unity Catalog (USE CATALOG, USE SCHEMA, SELECT, MODIFY on what the job needs) to groups and service principals, never to individual users, and never ALL PRIVILEGES.
• Make jobs idempotent and restartable: a rerun of the same input produces the same table (MERGE or replaceWhere by partition, never blind append).
• Validate bundles with databricks bundle validate in CI, where the pipeline signs in to the workspace; locally it needs a workspace and is reported as not run.

Conventions

Accessibility

Concurrency

• Delta transactions are optimistic: concurrent writers to the same table conflict, so design MERGE conditions and partitions so jobs touch disjoint data, and retry on ConcurrentAppendException with a bound.
• Set max_concurrent_runs: 1 on jobs that must not overlap, and use checkpoints per stream (never shared).

Dependencies

• Pin pyspark (development only), delta-spark, chispa and the job's libraries to versions matching the Databricks Runtime; commit a lock file or pinned requirements and audit them.
• Pin the Databricks CLI version in CI and the Databricks Runtime version (spark_version) of job clusters in the bundle.

Documentation

• Every table and column has a COMMENT in Unity Catalog (set by the code that creates it) and tables carry owner and domain tags.
• Docstrings on transformation functions say the input and output columns and the grain; the README explains the bundle targets, how to run the tests locally and how to deploy.
• Notebooks start with a markdown cell stating their purpose, parameters and outputs.

Errors

• Fail the job on bad data with explicit checks or expectations (expect_or_fail, assertions on counts and keys) that say which rule failed and how many rows; never swallow exceptions in a notebook cell.
• Configure retries, timeouts and failure notifications on job tasks in the bundle; make tasks idempotent so a retry is safe.
• Quarantine rows that fail validation to a separate table with the reason instead of dropping them silently.

Logging

• Log with the logging module (logger per module, structured key=value or JSON) to the driver log; record input and output row counts and the table versions read and written for each run.
• Use the job run's metrics, Delta table history (DESCRIBE HISTORY) and system tables (system.lakeflow, system.billing, system.access.audit) to monitor runs, cost and access; alert on failures through job notifications.
• Never log secrets (Databricks redacts secret values only in notebook output) or rows with personal data.

Naming

• Python modules, functions and variables in snake_case; DataFrame variables named for their content (orders, order_lines), never df1, df2.
• Tables and columns in snake_case, three-level names catalog.schema.table; schemas named for the layer or domain (bronze_sales, silver, gold_finance); no spaces or mixed case.
• Bundle resources named _ (orders_daily_job) and jobs' tasks for what they do; notebooks named for their step (01_ingest_orders.py).

Performance

• Cluster large Delta tables with liquid clustering (CLUSTER BY on the columns queries filter on) or, on older tables, partition by a low-cardinality date and Z-ORDER by filter columns; run OPTIMIZE and VACUUM on a schedule (or predictive optimization).
• Avoid collect(), toPandas() and Python UDFs on large data; prefer built-in functions, pandas UDFs where Python is needed, and broadcast joins only for small dimensions.
• Read only the columns and partitions needed, cache only DataFrames reused several times (and unpersist), and let adaptive query execution handle skew; check the Spark UI or query profile for spills and skew.
• Use incremental processing (Auto Loader, Structured Streaming with availableNow, MERGE of changed rows) instead of reprocessing all history.

Security

• Secrets come from secret scopes (dbutils.secrets.get(scope, key)) or the platform's identity; CI uses OAuth machine-to-machine with a service principal and workload identity federation; no personal access tokens or credentials in code, bundles, job parameters or notebooks.
• Unity Catalog grants to groups and service principals with least privilege (USE CATALOG, USE SCHEMA, SELECT, MODIFY on specific schemas), row filters and column masks for sensitive data; production jobs run_as a service principal.
• Validate external input (file schemas with an explicit StructType, rescued data columns, widget and job parameters against an allow-list) before it reaches SQL; build spark.sql statements with named parameters (spark.sql(query, args={...})), never f-strings with input (injection).
• Pin Python dependencies and audit them for vulnerabilities; install libraries on clusters from the bundle's built wheel or a pinned requirements file, never from an unpinned %pip install.

Structure

• databricks.yml at the root, resources/*.yml for jobs and pipelines, src// for the importable code (built into a wheel when jobs use it as a library), notebooks/ or src/notebooks/ for thin entry points, tests/ for pytest.
• Notebooks are committed in source format (.py with # Databricks notebook source, or .sql), never as .ipynb with outputs or .dbc archives.
• Configuration (catalog, schema, paths) comes from bundle variables, job parameters or widgets read once at the entry point and passed to functions as arguments.
• pyproject.toml declares the package and its dependencies; pyspark is a development dependency matching the cluster's Databricks Runtime Spark version, never installed onto the cluster.

Testing

• Unit tests with pytest and a local SparkSession for every transformation function: small input DataFrames with explicit schemas, assertDataFrameEqual (pyspark.testing) or chispa assert_df_equality, including nulls, duplicates and empty inputs.
• Integration tests run the bundle's job in a dev or staging workspace (databricks bundle run) against test data in a separate catalog; they need a workspace and run in CI, not on a developer's machine.
• Name tests test__ (test_add_order_totals_ignores_cancelled_lines) and keep one behaviour per test.
• Done means tested: new or changed behaviour ships with tests in the same change, and a fixed bug gets a regression test with the failing input.
• Keep tests fast: one shared session fixture, local[1], shuffle partitions 1, and no Delta or network I/O unless the test is about it (use a temporary directory for Delta with delta-spark).

Review checklist

• Is the logic in importable functions with unit tests on a local SparkSession?
• Do the tests cover nulls, duplicates and empty inputs, comparing DataFrames with schema?
• Are tables addressed by three-level names with the catalog from configuration?
• Are writes idempotent (MERGE or replaceWhere), with explicit schemas?
• Is there no collect(), toPandas() or Python UDF on large data?
• Are large tables clustered (liquid clustering, or partitioning with Z-order) for their queries?
• Are secrets read from secret scopes, with no token or password in code, notebooks or bundle files?
• Are Unity Catalog grants least privilege, to groups or service principals?
• Are jobs, pipelines and permissions defined in the Asset Bundle, with retries and notifications?
• Are notebooks committed in source format without outputs?
• Do ruff and the type checker pass, and are dependencies pinned to the runtime's versions?
• Are tables and columns documented with comments?

Quality checks (run when your repository has the tool configured)