Streamlining Data Operations with a Metadata-Driven Data Lakehouse on Azure
DS Stream migrated a Fortune 500 FMCG company's Azure Databricks data lake, which suffered from complexity, duplicated datasets and no clear structure, to a metadata-driven data lakehouse built on the medallion architecture. The result: better data discoverability and usability, automated pipelines that significantly reduced pipeline costs, and Great Expectations checks that keep data reliable.
Results at a glance
- Data discoverability and usability improved substantially after the move to a metadata-driven data lakehouse.
- Automated data extraction, archiving and incremental loads significantly reduced pipeline costs and raised operational efficiency.
- The medallion structure lets citizen developers work with datasets directly, which supports self-service analytics.
- Automated validation with Great Expectations protects data integrity and reliability.
- The new structure was enforced without disrupting existing user workflows.
What problem did the FMCG company have with its Azure Databricks data lake?
The client, a Fortune 500 FMCG company, ran an Azure Databricks data lake that had become complex, held duplicated datasets and lacked structure. The team needed a simpler way to run data operations, with better data quality and easier data discovery, and it had to keep costs under control at the same time.
The goals set at the start of the project:
- Simplify day-to-day data operations
- Improve data quality
- Make data easier to find and use
- Optimize costs
What did DS Stream build to replace the data lake?
DS Stream built a metadata-driven data lakehouse on Azure that follows the medallion architecture on Databricks. A metadata-driven framework automates the data pipelines, the medallion layers give every dataset a clear place, and Great Expectations runs automated data quality checks. The solution is scalable and organized.
Key components
- Metadata-driven pipeline framework with automatic data extraction, archiving and incremental load support
- Medallion structure enforcement that stays flexible and easy for users to work with
- Great Expectations integration for automated data quality checks and validation
Technologies
Databricks, Python, Azure, Spark, CI/CD (Azure DevOps / GitHub)
How was the migration delivered without disrupting users?
The team migrated the existing Azure Databricks data lake to the new lakehouse while users kept working. The medallion structure was enforced through the framework rather than through manual rules, so people kept a flexible, familiar way of working. CI/CD with Azure DevOps and GitHub supported delivery.
The project was delivered as a Data Engineering engagement. Because the framework handles structure and quality in the background, users could keep their workflows while datasets moved into a clear, governed layout.
Before and after the metadata-driven data lakehouse
| Area | Before | After |
|---|---|---|
| Structure | Complex data lake without a clear structure | Medallion architecture enforced by a metadata-driven framework |
| Datasets | Duplicated datasets | Organized, scalable lakehouse |
| Data quality | Quality needed improvement | Automated checks and validation with Great Expectations |
| Discoverability | Hard to find and use data | Substantially better discoverability and usability |
| Pipeline operations | Costs needed optimizing | Automated extraction, archiving and incremental loads with significantly lower pipeline costs |
| Data access | Not described in the source | Citizen developers engage with datasets directly |
Questions about this project
What business impact did the data lakehouse deliver?
The project streamlined the client's data processes and data management. The scalable solution helps the company make fuller use of its data assets, and it supports self-service analytics, operational cost savings and faster decision-making in a competitive FMCG market.
Why does the medallion architecture matter for self-service analytics?
In this project, the medallion architecture gave the lakehouse a clear and predictable structure. That clarity made datasets easier to find and understand, so citizen developers could work with the data directly instead of waiting for engineers, which supports self-service analytics and innovation.
How does the lakehouse keep data reliable?
Great Expectations is integrated into the pipelines and runs automated data quality checks and validation. This ensures data integrity and reliability and helps the platform meet the client's high quality standards without manual review of every load.
Related services and case studies
- Data Lake ArchitectureThe service behind this project: designing lakehouse structures that stay organized as data grows.
- Databricks solutionsHow DS Stream works with Databricks, the core platform of this Azure lakehouse.
- AI Data QualityAutomated data quality checks, the role Great Expectations plays in this case.
- Data engineering for FMCGOther data and AI work for FMCG and CPG companies like this client.
- Cloudera to GCP migration for a global CPG leaderAnother data platform migration for a consumer goods company, on Google Cloud.
- All case studiesBrowse DS Stream projects across data engineering, MLOps and AI.
Is your data lake getting harder to manage?
Tell us about your data platform. Dominik Radwański, Service Delivery Partner, and the team will get back to you within 4 hours on working days (Mon – Fri, 9am – 5pm CET).
Contact DS Stream