Uploaded November 2025 | Updated November 2026, 1 day ago
This is an end end to data engineering project using databricks free edition that is perfect for beginners as well as advanced users. In FMCG domain, we are building an ETL pipeline for a real life industry use case where a big retail company acquires a small one and we will build the data pipeline to consolidate data from both the companies in a single lakehouse architecture.
The tech stack data engineering project is: Python, SQL, Amazon S3, Medallion architecture, Spark, BI dashboard and Genie.
Signup to Databricks Free Edition: bit.ly/4nK0NTN
Databricks foundations video: youtube.com/watch?v=761SQ9Hxbic
Code, dataset and other resources: resources.codebasics.io/pD0lLR
๐ Ready to go from Data Analyst to AI-Enabled Data Engineer? Our live bootcamp takes you from dashboards to production pipelines: resources.codebasics.io/5yaUOR
๐ฎ Want to learn by doing? Datakata is a free, super fun game where you build data pipelines hands-on: resources.codebasics.io/Ahkp3O
Get Certified by Codebasics with this Databricks Mini Course : resources.codebasics.io/dgbaAX
โญ๏ธ Timestamps โญ๏ธ
00:00:00 Intro
00:00:57 Project Introduction
00:02:57 Stakeholder Discussion
00:04:22 2 Learning Paths
00:04:40 Tech Architecture
00:09:47 Data Model
00:16:42 Signup to Databricks free edition
00:17:26 Setup Catalog. Data Import
00:27:36 AWS Account Setup
00:30:48 Upload Data to S3
00:32:50 Databricks S3 Connection
00:34:34 Dimension Data Processing
01:40:23 Fact Processing - Historical Load
02:02:11 Fact Processing - Incremental Load
02:25:51 Orchestration
02:38:07 Denormalized Table and Genie
02:43:24 Dashboarding
02:54:46 Stakeholder Review Meeting
02:55:55 Conclusion
Do you want to learn technology from me? Check codebasics.io/?utm_source=description&utm_medium=yt&utm_campaign=description&utm_id=description for my affordable video courses.
Need help building software or data analytics/AI solutions? My company atliq.com can help. Click on the Contact button on that website.
๐ฅ Codebasics Hindi channel: youtube.com/channel/UCTmFBhuhMibVoSfYom1uXEg
#๏ธโฃ Social Media #๏ธโฃ
๐งโ๐คโ๐ง Discord for Community Support: discord.gg/r42Kbuk
๐ธ Codebasics' Instagram: instagram.com/codebasicshub
๐ Codebasics' Linkedin : linkedin.com/company/codebasics
------
๐ Dhaval's Linkedin : linkedin.com/in/dhavalsays
๐ Hem's Linkedin: linkedin.com/in/hemvad
๐ฝ๏ธ Hem's Instagram for daily tips: instagram.com/hemvadivel
๐ธ Dhaval's Personal Instagram: instagram.com/dhavalsays
๐ Patreon: patreon.com/codebasics?fan_landing=true
This is an end end to data engineering project using databricks free edition that is perfect for beginners as well as advanced users. In FMCG domain, we are building an ETL pipeline for a real life industry use case where a big retail company acquires a small one and we will build the data pipeline to consolidate data from both the companies in a single lakehouse architecture.
The tech stack data engineering project is: Python, SQL, Amazon S3, Medallion architecture, Spark, BI dashboard and Genie.
Signup to Databricks Free Edition: bit.ly/4nK0NTN
Databricks foundations video: youtube.com/watch?v=761SQ9Hxbic
Code, dataset and other resources: resources.codebasics.io/pD0lLR
๐ Ready to go from Data Analyst to AI-Enabled Data Engineer? Our live bootcamp takes you from dashboards to production pipelines: resources.codebasics.io/5yaUOR
๐ฎ Want to learn by doing? Datakata is a free, super fun game where you build data pipelines hands-on: resources.codebasics.io/Ahkp3O
Get Certified by Codebasics with this Databricks Mini Course : resources.codebasics.io/dgbaAX
โญ๏ธ Timestamps โญ๏ธ
00:00:00 Intro
00:00:57 Project Introduction
00:02:57 Stakeholder Discussion
00:04:22 2 Learning Paths
00:04:40 Tech Architecture
00:09:47 Data Model
00:16:42 Signup to Databricks free edition
00:17:26 Setup Catalog. Data Import
00:27:36 AWS Account Setup
00:30:48 Upload Data to S3
00:32:50 Databricks S3 Connection
00:34:34 Dimension Data Processing
01:40:23 Fact Processing - Historical Load
02:02:11 Fact Processing - Incremental Load
02:25:51 Orchestration
02:38:07 Denormalized Table and Genie
02:43:24 Dashboarding
02:54:46 Stakeholder Review Meeting
02:55:55 Conclusion
Do you want to learn technology from me? Check codebasics.io/?utm_source=description&utm_medium=yt&utm_campaign=description&utm_id=description for my affordable video courses.
Need help building software or data analytics/AI solutions? My company atliq.com can help. Click on the Contact button on that website.
๐ฅ Codebasics Hindi channel: youtube.com/channel/UCTmFBhuhMibVoSfYom1uXEg
#๏ธโฃ Social Media #๏ธโฃ
๐งโ๐คโ๐ง Discord for Community Support: discord.gg/r42Kbuk
๐ธ Codebasics' Instagram: instagram.com/codebasicshub
๐ Codebasics' Linkedin : linkedin.com/company/codebasics
------
๐ Dhaval's Linkedin : linkedin.com/in/dhavalsays
๐ Hem's Linkedin: linkedin.com/in/hemvad
๐ฝ๏ธ Hem's Instagram for daily tips: instagram.com/hemvadivel
๐ธ Dhaval's Personal Instagram: instagram.com/dhavalsays
๐ Patreon: patreon.com/codebasics?fan_landing=true










