Feature Store Optimization for Kpler: Ingestion Cut from Four Hours to Ten Minutes
Kpler, a leader in market intelligence and data analytics, had slow Feature Store ingestion, database connection issues and little observability in model training. DS Stream optimized the ingestion process, redesigned the Feature Store framework and hardened the model execution workflows. Data processing time fell from four hours to ten minutes, and the database connection issues were eliminated.
Key results
- Feature Store ingestion time dropped from four hours to ten minutes.
- Database connection issues were eliminated, so operations became more stable.
- A new, standardized Feature Store framework made model integration simpler for MLOps and Data Science teams.
- Model execution became traceable, with no more log loss.
- Kpler extended the partnership to further refine and automate its model training processes.
What was slowing down Kpler's Feature Store and model execution workflows?
Kpler's Feature Store ingestion took too long, database connections failed under load, and model training gave the team little visibility. These problems made the workflows resource-intensive. They also created operational bottlenecks that slowed real-time, data-driven decisions across the business.
The main pain points were:
- Long data processing times in Feature Store ingestion
- Database connection issues
- No observability in model training
- Resource-intensive workflows and operational bottlenecks
How did DS Stream deliver the optimization?
DS Stream worked in three phases. Each phase built on the one before it.
Review and stabilize ingestion
We reviewed the existing ingestion process end to end and found the inefficiencies. We added monitoring tools to track data flows. Together with Kpler's DevOps team, we fixed connection pool exhaustion and set up a read replica. Unit tests and failure management mechanisms made the process more reliable.
Redesign the Feature Store framework
We analyzed how the Feature Store API worked with different models and proposed changes to improve efficiency and scalability. A new framework gave better integration with the data pipelines and simpler access to models.
Validate model execution
We validated the full model execution process. We added solutions that prevent log loss and optimized the training workflows to improve traceability and performance.
Which services and technologies did the Kpler project use?
The project combined two DS Stream services, MLOps and Data Engineering. It ran on a technology stack of AWS, Kubernetes, EKS and ARGO CD. Ingestion, the Feature Store framework and model execution workflows were all optimized as one engagement.
- Services: MLOps, Data Engineering
- Technologies: AWS, Kubernetes, EKS, ARGO CD
Measured result
10 minutes
Feature Store ingestion processing time after the optimization. Before the project, it took four hours.
Source: DS Stream case study: Optimizing Feature Store and Model Execution for Kpler
How did Kpler's workflows change?
| Area | Before | After |
|---|---|---|
| Data processing time | Four hours | Ten minutes |
| Database access | Connection pool exhaustion | Connection issues eliminated, read replica in place |
| Observability | Lacking in model training | Monitoring of data flows and traceable model execution |
| Logs | Log loss during model execution | Log loss prevented |
What did the new setup change for Kpler's data teams?
The standardized Feature Store framework gave MLOps and Data Science teams one shared way to integrate models, which improved how they worked together. Clear documentation and better observability made day-to-day operations smoother. They also supported more workflow automation.
The result was a data processing system that is more efficient, scalable and transparent. With faster and more reliable infrastructure, Kpler can generate insights sooner and act on them faster.
FAQ
What business impact did the Kpler project have?
The optimization brought substantial cost savings and better resource utilization, so Kpler could run more efficiently. A more reliable and scalable infrastructure lets the company generate insights faster. That supports better decisions and greater business agility.
Did Kpler continue working with DS Stream after the project?
Yes. After the results of this project, Kpler extended its partnership with DS Stream. The next work focuses on further refining and automating Kpler's model training processes, building on the Feature Store and model execution improvements described here.
Related pages
- Global Trade IntelligenceSee how DS Stream supports data-intensive companies in the industry Kpler works in.
- MLOps servicesLearn about the MLOps service behind the Feature Store framework and model execution work.
- Data Engineering servicesLearn about the data engineering work behind the faster, more stable ingestion.
- MLOps Platform for Scaling Deep Learning Training and InferencingAnother case study about scaling model training and inference.
- Transforming ML Workflow Through Standardization and AutomationA related case study on standardizing and automating ML workflows.
- All case studiesBrowse more DS Stream projects across industries.
Is your Feature Store or model execution slowing you down?
Tell us where your ingestion or model training gets stuck. Dominik Radwański, Service Delivery Partner, and our team will get back to you within 4 hours on working days (Mon – Fri, 9am – 5pm CET).
Contact DS Stream