Uploaded March 2026 | Updated September 2026, 2 weeks ago
Ryan Cicak, a solutions architect with Databricks, demonstrates how to use the Apache Iceberg REST Catalog (IRC) built into Unity Catalog (UC). He shows that any Apache Spark engine can easily connect to the IRC to read and write Spark tables managed by Unity Catalog. The demonstration uses Spark SQL running locally on a MacBook and connects to the catalog using an OAUTH 2 token.
🎯 What You'll Learn:
- How the Apache Iceberg REST Catalog (IRC) within Unity Catalog allows any engine to read and write managed Spark tables.
- How to configure Spark SQL with a connection string and an OAUTH 2 token to connect to the UC Iceberg REST Catalog.
- How to use a local, open-source Spark SQL instance to query tables managed in Databricks Unity Catalog.
- How to create new tables and insert data into Unity Catalog using a third-party OSS Spark engine.
⏱️ CHAPTERS:
00:00 – Introduction to the Apache Iceberg REST Catalog (IRC) and Unity Catalog
01:08 – Architecture and Connection String for IRC using an OAUTH 2 Token
02:40 – Preparing the Local Demo Environment and Viewing Existing Tables in Databricks
03:46 – Connecting Local Spark SQL to Unity Catalog and Querying Existing Tables
05:14 – Creating a New Table in Unity Catalog via OSS Spark SQL
06:17 – Inserting Data into the New Table and Final Verification
Ryan Cicak, a solutions architect with Databricks, demonstrates how to use the Apache Iceberg REST Catalog (IRC) built into Unity Catalog (UC). He shows that any Apache Spark engine can easily connect to the IRC to read and write Spark tables managed by Unity Catalog. The demonstration uses Spark SQL running locally on a MacBook and connects to the catalog using an OAUTH 2 token.
🎯 What You'll Learn:
- How the Apache Iceberg REST Catalog (IRC) within Unity Catalog allows any engine to read and write managed Spark tables.
- How to configure Spark SQL with a connection string and an OAUTH 2 token to connect to the UC Iceberg REST Catalog.
- How to use a local, open-source Spark SQL instance to query tables managed in Databricks Unity Catalog.
- How to create new tables and insert data into Unity Catalog using a third-party OSS Spark engine.
⏱️ CHAPTERS:
00:00 – Introduction to the Apache Iceberg REST Catalog (IRC) and Unity Catalog
01:08 – Architecture and Connection String for IRC using an OAUTH 2 Token
02:40 – Preparing the Local Demo Environment and Viewing Existing Tables in Databricks
03:46 – Connecting Local Spark SQL to Unity Catalog and Querying Existing Tables
05:14 – Creating a New Table in Unity Catalog via OSS Spark SQL
06:17 – Inserting Data into the New Table and Final Verification
