Space Apps Challenge is an annual global contest where scientists and innovators from across the world compete to solve challenges facing the aerospace industry proposed by NASA. In 2016, I was one of the scientists that won this prize thanks to an algorithm capable of detecting and classifying Near Earth Objects, or NEOs. Today, I work as a data scientist at BBVA, where I develop fraud detection models. This is my story.
This post in 10 seconds
In this post we will address software dependency management: the security problems it implies, how to automate its management, receiving new vulnerabilities alerts in real time and how to implement a productive and satisfactory system using new Open Source tools as Deeptracy and Patton.
There are a few Container Orchestration Engines available, such as Kubernetes, OpenShift, DC/OS, Nomad or Swarm, to name a few. On occasion, the optimal solution to deploy your apps requieres picking not one of those, but two or more. Each have their strenghts, and are targetted toward a specific kind of load. For instance, Kubernetes is specially prepared to deploy microservices very dynamically. However, in such a dynamic environment it is hard to handle a persistent database, task which is better handled by system such as DC/OS.
At BBVA Labs, we have been using Akka Actors in different projects for a long time due to its inherent computation distribution capabilities in high load scenarios, using actors as stateless or stateful units of execution that send messages among them in an asynchronous fashion.
In this post, we will cover some of the weak points of working with untyped classic actors and we will show what are the benefits that Akka Typed brings to the scene. Finally, we will present a brief description of session types and the Process DSL API created by Dr. Roland Kuhn. All examples are written using the Scala API.
Deep Learning techniques are providing amazing results on several Machine Learning fields, but training a complex deep neural network on a large dataset can take a significant amount of time. Therefore accelerate training of Deep Learning models is really crucial, but also challenging.
Today, a large majority of modern apps or services are RESTful APIs and use API definitions to facilitate communications between them, as it saves us from having to worry about the language and the underlying implementation of the remaining components. APIs make even more sense in microservice or serverless architectures with dozens or hundreds of mutually interacting microservices/functions.
At BBVA Labs, we follow the test pyramid concept proposed by Mike Cohn. We have a large collection of unit tests that are easy to implement and which are run at every change in code; a collection of acceptance tests which are run whenever the previous tests are passed; and finally, end-to-end tests that are only run to unlock a function.
The complexity of implementing these tests increases as you move up in the pyramid. End-to-end tests, where service integration is also tested, requires putting in place the infrastructure, the services to be tested and the integrated services. Setting up a testing environment, test implementation and execution are significantly more complex than unit tests.
In addition to the cost of running these tests, another problem arises when a service changes the message format. Big bang deployment (deploying the service and its dependents at the same time) is to be avoided as this type of change breaks the continual deployment) . Therefore, for a period of time, the service provider has to offer support for two versions of the message while customers update to the new one, but consumer tests only look at one version of the producer.
At BBVA Labs, we conducted an experiment to reduce the number of services to deploy in tests and to ensure that communication among services from different domains is maintained throughout the software product’s life cycle in a continual deployment system. The decision was made to evaluate current tools to conduct Consumer Driven Contract testing (CDC testing o contract testing) for this experiment.
Using Docker in the deployment of software in productive systems solves many problems related to agility and the normalization of these processes. But, like all technology that breaks with prior IT processes, it generates new challenges or requires different solutions for persisting problems. One of those is the management of secrets.
This article is the second part about serverless, where we will cover the integration of one of the most interesting products implementing this technology (Fission) in OpenShift, RedHat’s PaaS platform.
Prior to this article, this analysis of serverless architectures or FaaS (Function as a Service) was prepared, to introduce this technology together with its value in business developments.
Serverless architecture, also known as FaaS (Functions as a Service) enables the execution of an application via ephemeral, stateless containers; these are created at the moment an event is produced that sets off this application. Contrary to what the term suggests, serverless does not mean “without a server.” Rather, these are used as an anonymous element of the infrastructure, supported by the advantages of cloud computing.
Here, leaving aside the hype that accompanies the term serverless, we explore in this post the possibilities it offers in architecture and applications development. We also study the principal alternatives for use in public and private clouds.
Neural networks training is a time consuming activity, the amount of computation needed is usually high even for today standards. There are two ways to reduce the time needed, use more powerful machines or use more machines.
The first approach can be achieved using dedicated hardware like GPUs or maybe FPGAs or TPUs in the future. But it can also be done by splitting the task between more general purpose hardware like the one used in cloud systems.
This document summarizes the conclusions reached after researching the use of distributed neural networks.