MiniLake Is a Local Databricks API Emulator with SQL, Delta Lake and Spark

MiniLake is a local Databricks API emulator with real SQL, real Delta Lake and real Spark job execution. Point databricks-sdk, the Databricks CLI, Terraform, Asset Bundles or an LLM agent at http://localhost:8000 - no account, no API key, no cloud bill.

Installation

pip install minilake && minilake --port 8000  # PyPI
docker run -p 8000:8000 ghcr.io/dmux/minilake:latest  # GHCR

Then open http://localhost:8000/ui/ for the built-in SQL workspace. The image downloads nothing at runtime - it works air-gapped.

Fake endpoints let broken code pass. MiniLake executes your SQL on DuckDB, writes real Delta files, and runs real spark-submit jobs in containers - so a green test locally means something.

Unity Catalog, for real

Catalogs, schemas, tables and volumes with full CRUD. Every catalog is its own DuckDB database file, so catalog.schema.table addressing is native - not a regex rewrite.

Statement execution on real DuckDB: INLINE and EXTERNAL_LINKS dispositions, JSON_ARRAY / ARROW_STREAM / CSV formats, and a result manifest that carries true column types.

EXTERNAL Delta tables are genuine Parquet plus a _delta_log/. INSERT, UPDATE and DELETE are routed through a real Spark job; reads go through DuckDB's delta_scan().

spark_python_task runs spark-submit in a sibling container and returns actual stdout. Real DAG scheduling honours depends_on and run_if, with independent branches in parallel.

spark.table("cat.sch.tbl") resolves against MiniLake through the official UCSingleCatalog connector - the same protocol Spark speaks to a real workspace.

An Athena-style editor at /ui: Monaco with catalog-aware completion, a virtualized result grid, charts, CSV/JSON export, saved queries, query history - and an embedded JupyterLab.

Point the databricks provider at localhost and terraform apply creates real catalogs. Verified on every test run: the real provider is driven through apply, plan and destroy, and a second plan has to come back empty.

databricks bundle deploy and bundle run work end to end - jobs execute for real.

67 tools, 8 resources and 4 prompts at /mcp - including composites that collapse a five-call sequence into one, and a SQL-dialect resource that stops agents writing Spark SQL.

One POST /_minilake/reset gives every test a clean workspace without restarting the container. /health, /ready and /services round out the control plane.

File-backed notebook and file storage with chunked uploads. Secrets are real CRUD whose values are only ever resolvable inside a job's environment - exactly like Databricks.

Real execution

DuckDB's delta extension and the Delta / Unity Catalog jars are baked in at build time. The image downloads nothing at runtime - provable with docker run --network none.

Failures come back as {"error_code", "message"}, so the SDK raises its normal typed exceptions. Anything out of scope returns a loud 501 NOT_IMPLEMENTED instead of a convincing lie.

MiniLake speaks the Databricks REST API on the wire, so every client you already use keeps working - the same calls, the same response shapes, the same typed errors.

STRING, LONG, DECIMAL(10,2), ARRAY, MAP and STRUCT map to DuckDB and come back as the SDK's ColumnInfo. MINILAKE_PERSIST=1 snapshots state on shutdown and restores it on start.

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.catalog import ColumnInfo, TableType

# The only line that differs from production
w = WorkspaceClient(host="http://localhost:8000", token="dev")
w.catalogs.create(name="sales")
w.schemas.create(name="store", catalog_name="sales")
w.tables.create(
    name="orders",
    catalog_name="sales",
    schema_name="store",
    table_type=TableType.MANAGED,
    data_source_format="DELTA",
    storage_location="/data/sales/store/orders",
    columns=[
        ColumnInfo(name="id", type_text="BIGINT"),
        ColumnInfo(name="customer", type_text="STRING"),
        ColumnInfo(name="amount", type_text="DECIMAL(10,2)"),
    ],
)
wh = w.warehouses.create(name="dev", cluster_size="Small")
w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="INSERT INTO sales.store.orders VALUES (1, 'Ana', 459.90)",
)
result = w.statement_execution.execute_statement(
    warehouse_id=wh.id,
    statement="SELECT customer, amount FROM sales.store.orders",
)
print(result.result.data_array)  # [['Ana', '459.90']] - really stored

from pyspark.sql import SparkSession
UC = "http://localhost:8000"
spark = (
    SparkSession.builder
    .config("spark.sql.extensions", "io.delta.sql.DeltaSparkSessionExtension")
    # Unity Catalog connector - three-part names resolve against MiniLake
    .config("spark.sql.catalog.spark_catalog", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.spark_catalog.uri", UC)
    .config("spark.sql.catalog.sales", "io.unitycatalog.spark.UCSingleCatalog")
    .config("spark.sql.catalog.sales.uri", UC)
    .getOrCreate()
)
# Real Delta files: parquet + _delta_log/, written by real Spark
spark.createDataFrame([(1, "a"), (2, "b")], ["id", "name"]) \
    .write.format("delta").mode("overwrite").save("/data/delta/sales/store/events")
spark.sql("INSERT INTO sales.store.events VALUES (3, 'c')")
print(spark.table("sales.store.events").count())  # 3
# ...and the same bytes are readable through MiniLake's SQL API:
# SELECT e.name, o.customer
# FROM sales.store.events e JOIN sales.store.orders o ON o.id = e.id

provider "databricks" {
  host = "http://localhost:8000"
  token = "dev"
}
resource "databricks_catalog" "sandbox" {
  name = "sandbox"
  comment = "Local sandbox catalog"
}
resource "databricks_schema" "things" {
  catalog_name = databricks_catalog.sandbox.name
  name = "things"
}
resource "databricks_sql_endpoint" "compute" {
  name = "local-compute"
  cluster_size = "Small"
}
# terraform apply -> creates real catalogs and real DuckDB databases
# terraform destroy -> removes them
# Resources backed by APIs MiniLake does not emulate fail loudly with 501,
# instead of silently pretending to succeed.

bundle:
  name: my_pipeline
targets:
  dev:
    mode: development
    workspace:
      host: http://localhost:8000
resources:
  jobs:
    daily_load:
      name: daily_load
      tasks:
        - task_key: main
          spark_python_task:
            python_file: ../src/load.py
# $ databricks bundle deploy -t dev
# $ databricks bundle run daily_load -t dev
#
# The CLI syncs files through the Workspace API, the provider creates the job,
# and `bundle run` executes it on real Spark in a sibling container -
# returning the job's actual output.

# Is it up, and which services answered?
curl http://localhost:8000/_minilake/health
# Create a catalog through the Databricks API surface
curl -X POST http://localhost:8000/api/2.1/unity-catalog/catalogs \
  -H 'Content-Type: application/json' \
  -d '{"name": "sales"}'
# Run SQL and get real rows back
curl -X POST http://localhost:8000/api/2.0/sql/statements \
  -H 'Content-Type: application/json' \
  -d '{"warehouse_id": "dev", "statement": "SELECT 42 AS answer"}'
# Give the next test a clean workspace - no restart needed
curl -X POST http://localhost:8000/_minilake/reset

Architecture

A FastAPI application serves the Databricks REST surface, the web UI and the MCP endpoint together. Behind it sit DuckDB, real Delta files on disk, and Docker for Spark.

A Databricks workspace is an awkward thing for an agent to drive: every useful action is four calls deep, half the SQL it writes is the wrong dialect, and failures are silent. MiniLake's MCP server closes those gaps - and it costs nothing when the agent gets it wrong.

Plus 8 resources - including a SQL-dialect guide that stops agents writing Spark SQL at DuckDB - and 4 prompts that front-load the rules. Tools run through MiniLake's own ASGI stack, so an agent's error is exactly the error an SDK client would have seen.

# One line to register it with Claude Code
claude mcp add --transport http minilake http://localhost:8000/mcp
# Then just ask:
# "Use minilake: create a sales catalog with a seeded
# orders table, and tell me the total."
# The agent does this - three calls, no glue code:
# setup_fixture({"catalog": "sales", "schema": "store"})
# seed_table({"full_name": "sales.store.orders", ...})
# run_sql({"statement": "SELECT SUM(amount) FROM sales.store.orders"})
# Enable it on the server side: MINILAKE_MCP=1 minilake --port 8000

Service status

Every row below is either really executed or clearly labelled. Anything out of scope answers 501 NOT_IMPLEMENTED rather than faking a success your production code will not get.

These are deliberate design decisions, not a backlog. MiniLake targets one developer running it locally - not a shared, multi-tenant server. Any token is accepted and there is exactly one user. Never expose it beyond localhost.

The Permissions API is real CRUD that always allows everything - a passing test here says nothing about grants in a real workspace. Statements run on DuckDB's dialect. The MCP server ships a dialect resource for exactly this reason; the docs carry the translation table.

The cluster lifecycle is a state machine. Real Spark happens through Jobs, in sibling containers. DuckDB is single-writer, so concurrent load serializes on a lock. Fine for dev and CI; not a deployment target.

They are DuckDB tables with no files. Use EXTERNAL Delta when Spark has to see the data - that trade-off is what keeps SQL fast.

Being explicit about the edges is the point: an emulator that quietly fakes what it cannot do is worse than no emulator at all.

Free and MIT-licensed. No account, no API key, no cloud bill - and nothing to uninstall from your cloud account when you're done.