MLOps transformation at BBVA: delivering business value faster

When designing our new MLOps architecture, we knew that building a theoretical framework would not be enough. To ensure our architectural decisions met real production requirements from the beginning, we designed our MLOps foundation working in parallel with four distinct business units. These initial pilots spanned highly diverse domains—from risk modeling and pricing optimization to personalized recommendations and financial forecasting—involving varying data scales, compute needs, and team maturity levels.

Despite operating across different geographies and domains, we discovered that the teams shared common opportunities to develop and standardize their ML delivery practices. While they already leveraged code repositories, experiment tracking, and model lifecycle capabilities, implementation patterns had naturally evolved independently over time, reducing reuse across teams and increasing onboarding effort for new projects with similar technical foundations.

Furthermore, operational processes around model promotion and pipeline management also combined automated and manual activities, creating opportunities to improve scalability as the number of use cases grew. The absence of isolated testing environments meant validating pipeline changes required merging to shared branches, slowing feedback loops. Similarly, although teams had essential governance capabilities in place—like model registration and monitoring—their approaches varied widely.

The opportunity was clear: accelerate and industrialize ML delivery by standardizing common patterns into reusable templates, simplifying operational workflows, and embedding governance and traceability directly into the development experience.

Method: Aligning People, Processes, and Technology

Modernizing Machine Learning workflows across a large organization requires more than introducing new tools; it requires defining a scalable approach that aligns people, processes, governance, and technology. To achieve this, we combined BBVA’s domain knowledge with AWS expertise and conducted a series of discovery and assessment sessions with several business units.

We kicked off with structured discovery sessions across each of the four business units that agreed to be early adopters of the solution. Each business unit brought distinct ML technical use cases (from XGBoost regression to Deep Neural Networks), different development maturity levels, and operational constraints. Rather than prescribing a single path, we used these sessions to map these diverse workflows to identify common patterns, challenges, and divergences.

This dual-track approach was crucial to align real-world business needs with the general MLOps platform being designed. The result was a shared understanding of what "good" looked like across the organization with real data points.

The findings from these discovery sessions fed directly into the design of reusable MLOps templates and standardized operational workflows. The resulting technical framework balanced standardization across four dimensions: guiding people to transition from individual notebook work to collaborative, version-controlled pipelines; establishing standardized processes for code review, testing, and deployment; embedding governance to ensure regulatory alignment and reduce manual oversight; and leveraging technology by integrating MLOps templates directly into ADA, the BBVA’s global Data and AI platform, so every new project begins with a production-ready foundation.

Solution Overview

The core of this industrialized approach begins in the ADA console. When users select an MLOps asset, the system automatically provisions a new project with a dedicated GitHub repository and a standardized code scaffold. This scaffold provides an end-to-end setup for the Machine Learning lifecycle, including reusable Amazon SageMaker Pipelines for training, inference, and monitoring, enabling teams to customize specific workflows while inheriting built-in governance, experiment tracking, and continuous integration capabilities.

The training pipelines are intentionally modular. The default template structures the build process into discrete steps: data processing, training, evaluation, performance-based condition checks, quality checks, and registration into the SageMaker Model Registry. This modular design enables step-level caching, which skips unchanged steps during iterative development to reduce execution time and compute costs. A CPU-first compute strategy ensures that expensive resources are utilized only when strictly necessary.

Additionally, the template promotes a Python-first approach that replaces PySpark where feasible, simplifying the development experience for use cases that do not require distributed compute, while still allowing teams to integrate EMR-backed PySpark steps when large-scale data preparation is necessary

Models are conditionally registered only if they meet predefined performance thresholds, while tools like MLflow capture hyperparameters, metrics, and artifacts to maintain a centralized tracking record mapped to the corresponding Model Registry version.

Similarly, the inference pipelines are structured to minimize manual coordination. They feature an automated retrieval mechanism utilizing an AWS Lambda step that seamlessly resolves the latest approved model version artifacts, eliminating the need to manage paths manually. When a retrained model completes the automated approval workflow without material changes to the underlying training pipeline, the inference setup retrieves the latest approved version automatically. This capability keeps production predictions up to date without requiring a new deployment or re-approval of the inference pipeline itself.

Extending the MLOps Template for Complex Workloads

While the default template successfully covers standard Machine Learning development patterns, different business domains often introduce highly specialized technical and operational requirements. Rather than creating disconnected, bespoke workflows, the reusable template foundation was extended to support specialized workloads without diverging from the standardized pipeline structure.

For instance, when business units need to train compute-intensive deep neural networks in parallel to select the best-performing candidate, the pipeline seamlessly adapts to utilize GPU-optimized instance families. The container environment shifts to a CUDA-enabled framework image, while the overarching orchestration and conditional registration logic remain completely unchanged. Crucially, this high-performance computing is applied exclusively to the training step, while data processing and evaluation continue to run on cost-effective CPU instances to preserve the core cost-optimization strategy. Likewise, for domains managing massive datasets, a hybrid execution model was implemented to combine the scalability of PySpark-based distributed processing with the simplicity and flexibility of Python-based ML development,

The fundamental principle across these extensions is consistency. The orchestration, conditional logic, and governance layers remain completely decoupled from the execution framework. Every pipeline execution, whether running PyTorch on a GPU or Scikit-Learn on a CPU, logs to the same tracking system, registers models through the identical governance workflow, and integrates seamlessly with the CI/CD automation. This validates that the template architecture can accommodate immense technical diversity without fragmenting into team-specific solutions. You can find further information about this solution and extended ML templates in this blogpost from AWS.

Lessons Learned and Best Practices

The pilot phase provided clear insights into how a standardized but flexible MLOps framework can accelerate development, improve efficiency, and strengthen governance. The key best practices identified include:

• Start from real use cases: Building the MLOps standardization from actual production use cases helped identify true common patterns while preserving the flexibility required for different data volumes, model architectures, and compute needs.
• Treat templates as accelerators, not constraints: A generic MLOps template provides a production-ready foundation for training, inference, data monitoring, and model monitoring pipelines, with CI/CD, experiment tracking, model registration, and governance capabilities already integrated. Teams can extend these pipelines when needed, while still following a consistent lifecycle model.
• Enable ephemeral validation environments: Improving the developer experience is essential. Pull-request-based ephemeral environments allow teams to validate changes safely before merging into shared branches. Combined with automated tests and security checks, this parallel development approach enables faster feedback loops, better collaboration, and improved code quality.
• Optimize pipeline resources at the step level: Treating an entire pipeline as a single compute requirement is inefficient. By assigning the right compute resources to each stage, using caching, and combining PySpark, Python, CPU, and GPU workloads where appropriate, teams were able to reduce execution times and improve cost efficiency.
• Embed governance directly into de ML lifecycle: The standard workflow handles model versioning, approval workflows, model risk management (MRM) integration, monitoring, and traceability. This reduces manual effort and ensures that compliance is a seamless part of the development process.

Conclusion

By starting from real production use cases, we have defined a common MLOps foundation that accelerates delivery while preserving the flexibility required by different business domains, technical patterns, and regulatory requirements. At BBVA, we will continue improving these capabilities and expanding adoption across more teams, geographies, and ML projects.

This new way of working is already helping our teams reduce fragmentation, improve reuse, and strengthen how they control, monitor, and operate ML solutions in production, achieving delivery time improvements by up to 75%. And, most importantly, it allows us to develop AI more efficiently, with the aim of offering a smarter, more proactive and personalized banking service to our customers.

Authors

Margarita García García, Oscar García Ramos,  Andrea Perez Isla (AWS Professional Services), Carlos Guerra (BBVA Tech), Raúl Díaz García (AWS Professional Services).