E-commerce case study

Data Migration to BigQuery With Governed, Hands-Free Data Flows for an E-commerce Leader

An e-commerce platform leader needed a data migration to BigQuery, which was already in use across its organization. The native BigQuery writer in the client's tool had limited options and could not meet its data governance requirements. DS Stream wrapped the writer in Python code using the GCP API and added the missing features. The data transfer flows now run without human intervention.

Talk to our team See more case studies

Key results

All required migration tasks were completed, and data now lands in Google Cloud BigQuery.
Data transfer flows operate without the need for human intervention.
Governance and other features missing from the native BigQuery writer were added in Python on top of the GCP API.
No new tool was added to the client's stack. Employees keep working with current data in the tool they already know.
The solution can be extended to cover future requirements.

Why was the native BigQuery writer not enough for this e-commerce data migration?

The client had already rolled out BigQuery successfully across the organization, so it was the natural target for the migration. The blocker was the native BigQuery writer built into the client's tool. It offered only limited options and did not provide the level of data governance the client required. The migration could not go ahead with the writer as it was.

How did DS Stream close the gap between the writer and the client's governance needs?

DS Stream did not replace the native writer. The team embedded it in Python code that calls the GCP API and implements the features the writer was missing. The client kept the integration it already had and gained the extra functionality it required for governance.

What the solution does

Keeps the native BigQuery writer as the core of the data transfer
Uses the GCP API from Python to add the missing functionality
Runs the data transfer flows without manual steps
Can be extended when new requirements appear

What did the Google Cloud BigQuery migration change for the business?

Moving the data to a platform the client already used avoided the cost and effort of adding a new tool to its existing stack. Employees still reach current data through the familiar tool they are proficient in, so they did not need to learn a new workflow. The data transfer flows run on their own.

Native writer alone vs the extended solution

How the Python and GCP API layer changed the BigQuery data transfer
Criterion	Native BigQuery writer alone	Writer wrapped in Python and the GCP API
Available options	Limited	Missing features implemented in code
Data governance	Below the client's required level	Missing governance features added
Data transfer flows	Migration not yet possible	Run without human intervention
Future requirements	Bound by the writer's built-in options	Solution can be readily expanded

Questions about this BigQuery migration

Did the client have to adopt a new tool for the migration?

No. BigQuery was already in use across the client's organization, and the native BigQuery writer was already part of the client's tool. DS Stream extended that existing setup with Python code instead of adding new software to the stack.

Which technologies did DS Stream use in this project?

The target platform was Google Cloud BigQuery. DS Stream wrapped the native BigQuery writer in Python code that uses the GCP API. That code adds the features and data governance capabilities the writer lacked on its own.

Can the solution handle new requirements later?

Yes. The Python layer around the native writer was designed so it can be readily expanded. New features can be added through the GCP API without replacing the writer or the tool employees already use.

Related work and services

Tell us about your current tools and governance needs. We reply within 4 hours on working days (Mon–Fri, 9am–5pm CET).

Contact DS Stream