MLOps transformation at BBVA: from data platform to scalable ML

In recent years, the growth in the number of AI and ML solutions developed within the bank, coupled with the diversification of use cases, has made it necessary to build on a strong cloud platform that enables higher levels of flexibility, scalability, and automation.

Once its global deployment across BBVA was completed, the ADA platform continues to grow and currently serves more than 6,500 users in the bank, providing centralized access to development environments, shared data assets, and production deployment capabilities. Now, it also offers standardized and reusable ML pipelines, which technical teams use to create and deploy models consistently across business units, while also enabling faster experimentation and delivery.

To implement this evolution of the MLOps framework, BBVA leverages AWS technologies that provide the technical foundation while supporting the bank’s high standards for governance, risk management, and auditability.

In this article, we will look at the main updates to ADA’s architecture that are driving this transformation, such as ephemeral development workflows and an integrated, event-driven governance model.

The challenge: industrializing MLOps while maintaining strict governance

ADA’s ML workflows were built on a solid foundation that enabled rapid prototyping through standardized scaffolding and notebook-based workflows. However, as AI adoption grew, the focus shifted towards improving modularity, increasing code reuse, and strengthening reproducibility and version control practices across teams. Aligning development patterns across business units became a natural step to ensure consistency and long-term sustainability.

At the same time, we needed a solution that could provide data scientists with greater autonomy to experiment and iterate, while maintaining the platform’s stability and governance standards.

From an operational perspective, development and governance processes were clearly defined and effective, but they operated as separate workflows. As a result, optimizing lifecycle management through greater automation and integration became a priority. In particular, consolidating related assets — training pipelines, models, and inference pipelines — into unified project structures and automating approval and audit processes would reduce coordination overhead and improve traceability.

To solve this, we moved from a monolithic approach to a loosely coupled, flexible, and fully automated ML development workflow based on Amazon SageMaker AI. By providing standardized components and self-service capabilities, teams can now focus on generating business impact rather than managing infrastructure orchestration.

Solution overview

The MLOps architecture provides the infrastructure and tooling required for Data Scientists and ML Engineers to build ML pipelines for training, batch inference, and monitoring, while maintaining governance and traceability.

Within ADA’s platform, a user selects an MLOps template asset from a catalog and triggers the creation of a new project. This deployment collects important regulatory information metadata, such as the model description, business owner, and model tier associated with the models that will be developed. Each project automatically provisions a dedicated GitHub repository with a standardized code scaffold for training models, registering them alongside metrics and metadata.

The platform enables a fully automated ML development workflow, where standardized building blocks and self-service capabilities allow teams to develop, test, and deploy ML solutions independently, while relying on a shared, production-ready foundation. This allows teams to focus on delivering business impact, rather than managing infrastructure and orchestration complexity. Instead of implementing end-to-end logic in a single step, users can compose reusable pipeline components, parameterize them, and iterate more efficiently.

Additionally, to reduce testing time, the solution deploys dedicated ephemeral development workflows to test new features before they are merged to the main branch.

The transition between development and production environments is managed through GitHub Actions, which are attached to the associated repository branches in each environment. Amazon EventBridge and AWS Lambda, meanwhile, follow a model approval workflow and register the models in the production environment once they have been approved. You can see the end-to-end workflow for implementing ML models in this blogpost from AWS.

Ephemeral development workflows

One of the most transformative innovations in BBVA's MLOps evolution has been the introduction of ephemeral development workflows. This approach redefines how Data Scientists and ML Engineers validate and iterate on their work by moving from merge-based validation to a branch-based model that supports faster experimentation while maintaining quality standards.

Before implementing ephemeral workflows, validating changes required merging code into shared branches (main branch), introducing friction and potential risks in the development cycle. It also resulted in the proliferation of unused pipelines due to a sequential versioning approach.

The ephemeral workflow implementation leverages GitHub Actions and AWS SageMaker to create isolated, temporary resources for each pull request. When a developer opens a Pull Request (PR), the CI/CD platform automatically provisions a complete set of resources –including training pipelines and model packages—within a dedicated sandbox environment. These isolated environments use a standard naming convention, allowing multiple teams and users to work simultaneously on different features without interfering with shared resources.

Pipeline versioning and resource management

Crucially, this flexibility does not lead to cloud waste. Our ephemeral workflow system implements a rigorous automated resource cleanup. When a PR is closed or merged, the CI/CD platform executes a multi-step destruction process: it stops running executions, deletes model versions, removes pipeline definitions and cleans up model package groups. This automated lifecycle management has become a core operational discipline at BBVA, directly contributing to our 40% to 55% reduction in costs across pilot use cases.

Furthermore, we optimized pipeline versioning so that new versions are only natively generated when the pipeline definition or underlying code actually changes, reducing the total number of pipeline resources while improving traceability. This consolidation improves quota management, helping teams reduce quota consumption and enabling precise rollback and comparison capabilities.

Parallel development and corporate integration

A key element of BBVA’s evolution is that MLOps workflows are integrated with the bank’s corporate validation framework. These MLOps-specific workflows have been integrated into BBVA's centralized repository of GitHub Actions, making them easily reusable across all MLOps projects throughout the bank and ensuring consistent quality while reducing effort for new projects.

This parallel development capability has been the primary driver behind the 20% to 75% reduction in development times we have observed across pilot use cases. Developers can now iterate rapidly, experiment freely and test complex scenarios with total confidence.

Model governance framework

Governance is embedded by design within the platform, rather than introduced as a separate control layer. Approval rules, separation of duties, risk-tier-based promotion and auditability are no longer handled as disconnected manual steps, but as part of the platform workflow itself. SageMaker Model Registry is integrated with the bank’s model governance framework to enforce these rules programmatically while maintaining a streamlined, controlled experience for Data Scientists.

The framework addresses three core challenges:

• Ensure that no Data Scientist can approve their own model (self-approval prevention).
• Automate promotion decisions based on model risk tier.
• Maintain a centralized audit trail of every model lifecycle event.

These controls ensure that all model lifecycle transitions are traceable, auditable, and aligned with BBVA’s internal risk management policies.

Approval workflows

Models registered in SageMaker Model Registry progress through four stages: Development, QA, PreProduction, and Production. Each stage, paired with a specific status (InProgress, PendingApproval, Approved, or Rejected), determines the allowed transitions and the responsible role for each action.

To manage these transitions securely without slowing down innovation, the governance framework implements different approval workflows, each addressing a different governance concern:

• Auto-approval workflow: Because production models frequently require retraining, requiring a full approval cycle each time would prevent teams from keeping models current. To address this, an auto-approval path automatically promotes new model versions if the model was previously approved for production and the underlying training pipeline has not materially changed. If the pipeline definition does change, the model must go through the full review cycle.
• Manual-based approval workflow: When auto-promotion conditions are not met, a Data Scientist requests QA approval by updating the model lifecycle from Development/InProgress to QA/PendingApproval. A different Data Scientist reviews the model and either promotes it to the next stage or rejects it. Once a model reaches the PreProduction stage, the system automatically evaluates whether the Business Owner approval step can be skipped based on the model’s risk tier. Lower-risk models bypass this step and are automatically promoted to Production, while higher-risk models require rigorous manual reviews by another Data Scientist.
• Notifications across the model lifecycle: One of the most important process improvements is the tighter integration between ADA and the Model Risk Management platform. By automatically notifying the risk portal of key lifecycle events—such as project initialization or production promotion—the system eliminates manual handoffs, improves traceability across the full model lifecycle, and keeps both platforms seamlessly aligned.

Conclusion

BBVA’s MLOps transformation demonstrates how modernizing ML practices can deliver tangible business value when applied to real use cases. Across four pilot initiatives, teams achieved a 20–75% reduction in development time and a 40–55% reduction in costs, depending on the use case and initial level of MLOps maturity.

These improvements were driven by a new MLOps operating model within BBVA: standardized project creation, ephemeral validation environments and automated CI/CD workflows allow teams to move faster while maintaining the level of control required in a regulated banking environment.

Beyond efficiency gains, the platform, based on AWS technologies, embeds governance, compliance, and traceability by design. Centralized governance and automated approval workflows reduce operational risk while ensuring alignment with regulatory requirements.

This transformation represents a shift toward treating Machine Learning as a first-class engineering discipline within BBVA, supported by platform capabilities that enable scale, consistency, and continuous innovation.

Authors

Natalia Sampietro, Leticia García Martín (BBVA Tech), Itziar Molina Fernandez (AWS Professional Services), Juan Asensio Sánchez (AWS Professional Services), Néstor Durán (AWS Professional Services)