FMCG case study

A Unified Google Cloud Data Platform Gives a Global CPG Leader a Holistic View of Each Customer in Near Real-Time

A global FMCG / CPG company wanted a holistic view of each customer, but data silos blocked it. DS Stream built a unified data lake on Google Cloud: more than 100 Composer pipelines, roughly 1.5PB of historical data migrated in 6 months, about 30% lower storage and compute costs, and queries at least 15 times faster than Hadoop.

Talk to our data engineers See more case studies

Results at a glance

Roughly 1.5PB of historical data migrated to Google Cloud in 6 months.
Data storage and computation costs fell by approximately 30%.
BigQuery processed similar queries at least 15 times faster than the previous Hadoop solution.
More than 100 pipelines created and scheduled in Composer, processing millions of daily events at petabyte scale.
On-time, error-free data now serves many internal applications and 50+ brands in 100+ markets.

What problem did the global CPG company need to solve?

The client's data sat in silos, so campaigns and planning had no single trusted source. The goal was one unified data lake that would act as the authoritative source of data. It had to be cost-effective, secure and dependable, follow data stewardship standards, and connect to several third-party data vendors.

The client set clear expectations for the new platform:

Remove data silos and build a unified data lake for campaigns and planning.
Keep the solution cost-effective, secure and dependable.
Follow data stewardship standards.
Integrate smoothly with multiple third-party data vendors.
Use scalable, fully managed ETL services available on Google Cloud Platform.

What did DS Stream build on Google Cloud?

DS Stream designed a full set of integration processes for data ingestion, transformation and storage. At the core are more than 100 pipelines created and scheduled in Composer. They run on DataProc clusters, land data in Google Cloud Storage buckets and serve analytics from BigQuery, with Secret Manager for security and Logging for monitoring.

Platform components

Composer - creation and scheduling of 100+ pipelines.
DataProc - clusters for large-scale processing.
Google Cloud Storage - buckets for stored data.
BigQuery - the analytics layer that replaced Hadoop for querying.
Data Transfer tools - migration of historical data bundles.
Secret Manager - protection of credentials and sensitive settings.
Logging - monitoring of the running processes.

How was 1.5PB of historical data moved in 6 months?

Over 6 months, the team migrated roughly 1.5PB of historical data bundles with specialized Data Transfer tools. In parallel, DS Stream restructured many existing ingestion processes and orchestrated streamlined pipelines on Google Cloud Platform. Logging tracked each process, so issues were visible while data volumes moved past a petabyte.

The delivery covered three streams of work:

Migrating historical data bundles with specialized Data Transfer tools.
Restructuring numerous ingestion processes for the new platform.
Orchestrating streamlined pipelines on Google Cloud Platform, with security handled by Secret Manager and monitoring by Logging.

Measured outcomes

~1.5PB
Volume of historical data migrated to Google Cloud.
Source: DS Stream case study
6 months
Time taken to complete the historical data migration.
Source: DS Stream case study
~30%
Reduction in data storage and computation expenses.
Source: DS Stream case study
15x
Minimum speed gain for similar queries in BigQuery compared with the previous Hadoop solution.
Source: DS Stream case study
100+
Pipelines created and scheduled in Composer.
Source: DS Stream case study

Hadoop to BigQuery migration: before and after

Changes reported in this case study after the move to Google Cloud Platform.
Area	Before	After
Data sources	Siloed data	Unified data lake as the authoritative source for campaigns and planning
Query engine	Hadoop	BigQuery, at least 15 times faster on similar queries
Storage and compute cost	Previous cost level	About 30% lower
Pipelines	Numerous ingestion processes to restructure	100+ pipelines orchestrated in Composer
Security and monitoring	Not described in the source	Secret Manager and Logging

Questions about this project

How did the platform improve data quality and security?

The implemented solution markedly improved data quality, availability and security. Secret Manager protects sensitive configuration, Logging monitors the processes, and the design follows the client's data stewardship standards. Data now arrives on time and without errors, which makes it a dependable basis for campaigns and planning.

Who uses the data from the new platform?

The platform feeds a large number of internal applications at the client. Punctual, error-free data delivery supports more than 50 brands that operate in over 100 markets, all drawing on the same unified data lake instead of separate, siloed sources.