Uploaded April 2026 | Updated September 2026, 2 weeks ago

---

For years now, Databricks has been charging for open source software, so we decided to build our own with the help of Lisa Cao, Open Source advocate at Databricks.

Check out Lisa’s Lakehouse at Home project here: github.com/lisancao/lakehouse-at-home

00:00 Intro
00:54 Setup
10:51 Base diagram
16:09 Catalogs
23:06 Setting up Spark
25:19 Clusters Scaling
27:58 Clusters Diagram
30:54 SDP
37:19 Scaling Diagram
38:17 Outro

Links to all the open source projects we discussed:
MinIO: github.com/minio/minio
SeaweedFS: github.com/seaweedfs
Delta: github.com/delta-io
Iceberg: github.com/apache/iceberg
UC: github.com/unitycatalog
Polaris: github.com/apache/polaris
Lakekeeper: github.com/lakekeeper
Gavatino: github.com/apache/gravitino
Spark: github.com/apache/spark
Trino: github.com/trinodb
Flink: github.com/apache/flink
DuckDB: github.com/duckdb
Airflow: github.com/apache/airflow
Docker: github.com/docker-library
Kubernetes: github.com/kubernetes
Jupyter Notebooks: github.com/jupyter
MLFlow: github.com/mlflow


#OpenSource
#DataEngineering
#Lakehouse
#Databricks
#OverArchitected