Flagging mobile app anomalies in real time at Capital One

"Volume is hard to detect, measure, and alert on," said Donald Gennetten, a data engineer at Capital One. "When you try to do calculations on volume anomalies, you quickly realize that you have too many distinct thresholds to calculate and maintain." The team needed an approach that scaled without heavy coding and constant tuning.

Capital One's data scientists built the solution on open source and platform engineering so it met the bank's governance requirements. They used H2O Sparkling Water to prototype and deploy models quickly, pairing H2O-3's in-memory algorithms with Apache Spark and a unified Python, Scala, and H2O Flow pipeline. Grid search automated hyperparameter tuning across the team's experiments.

The team started with a GLM, then moved to gradient boosting (GBM), which handled the trend, seasonality, and data filtering that traditional time series methods miss. GBM let them exclude bad data, such as traffic during known downtime, and add signals like payment due dates that drive predictable Friday spikes. The production pipeline stores static data in AWS S3, runs the anomaly detector on Amazon EC2, keeps time series data in InfluxDB, and visualizes volumes and anomaly bands in Grafana.

The models give the operations team a live view of actual mobile volume overlaid with the forecast and a timestamped anomaly band, so an engineer triaging an incident can see immediately where current traffic sits against normal and when a deviation began.

Anomaly alerts now routinely fire ahead of other alarms. In one case, a spike of more than 20,000 unexpected logins at 11:15pm triggered an anomaly alert that reached the incident response team at 11:17pm, four minutes before any other alarm. "This was a huge win for our customers, and for the teams who are responding to these types of incidents," said Gennetten.