Write to usContact
Case study · FMCG · Data Migration

Cloudera to GCP Migration: Faster, Easier-to-Maintain Pipelines for a Global CPG Leader

DS Stream completed a Cloudera to GCP migration for a global FMCG / CPG company. Data pipelines moved to DataProc, GCS and Composer. Python ad optimization code for linear TV campaigns now runs on Kubeflow in GKE. The result is better performance, easier maintenance, more reliable data and shorter development time for optimization jobs.

Talk to our teamSee more case studies

Results at a glance

  • The Cloudera data pipelines were successfully migrated to the GCP platform.
  • Reliable cloud-native services led to better performance, easier maintenance and improved data reliability.
  • Kubeflow hosted on GKE significantly reduced the development time for optimization jobs.
  • The BigQuery cache keeps response times fast.
  • Final optimization jobs run in a flexible, robust and cost-optimized environment.

What problem did the CPG company need to solve?

The company planned and purchased linear TV campaigns using data from many sources. Those sources held various semi-structured data types and had data quality problems. The goal was to make the campaigns more cost-efficient by building pipelines on Kubeflow services. These pipelines had to improve system performance, make data transformation more reliable and optimize Python-based advertising procedures.

What the new platform had to deliver

  • Better overall system performance
  • More reliable data transformation across semi-structured sources
  • An optimized way to run the Python ad optimization code
  • Higher cost efficiency in linear TV planning and purchasing

How did DS Stream move the pipelines from Cloudera to GCP?

DS Stream migrated the existing data pipelines to DataProc, GCS and Composer. The team then containerized the Python ad optimization code. This let the code run as scalable tasks on Kubeflow hosted in Google Kubernetes Engine (GKE), instead of being tied to one fixed environment.

Kubeflow pipelines and node pools manage job resources. Different scenarios need different hardware, so each workload gets resources that fit it. This keeps resource utilization efficient. For related delivery patterns, see our data pipeline architecture services.

How does the new setup run optimization jobs?

Optimization jobs run on GKE with Kubeflow and Docker images. Each job can run on a different code version and different hardware resources. Cloud Functions start the jobs, which streamlines how the team launches optimization runs. BigQuery cache provides fast responses when the data is queried.

Why this matters for the team

  • Developers can test and run several code versions side by side.
  • Hardware is matched to each scenario through node pools.
  • Starting a job no longer requires manual setup steps.
  • The new pipelines are built for cost-effectiveness and easy maintenance.

Which Google Cloud services make up the new platform?

The target architecture uses Google Cloud products and open-source tooling. Each component has a clear role:

  • DataProc - runs the migrated data pipelines
  • GCS - stores pipeline data
  • Composer - orchestrates the migrated pipelines
  • BigQuery - serves data, with its cache keeping response times fast
  • Google Kubernetes Engine (GKE) - hosts Kubeflow and the containerized jobs
  • Kubeflow - runs pipelines and manages job resources through node pools
  • Docker images - package the Python ad optimization code in different versions
  • Cloud Functions - start optimization jobs

Read more about our approach to cloud migration.

Planning to move data pipelines off Cloudera?

Tell us about your current platform. We get back to you within 4 hours on working days (Mon-Fri, 9am-5pm CET).

Contact DS Stream