Make with H2O ai: Accuracy Masterclass Part 1 - Choosing the "Right" Metric for Success

Name: Make with H2O ai: Accuracy Masterclass Part 1 - Choosing the "Right" Metric for Success
Uploaded: 2022-07-26
Duration: 48 min
Description: A masterclass on choosing the right evaluation metric: regression metrics from RMSE to RMSLE, and classification metrics, from a Kaggle Grandmaster.

Predictive AI
H2O Driverless AI

Overview

This masterclass, led by a presenter with Kaggle grandmaster level experience across more than 200 competitions, breaks down how to choose the right evaluation metric for a machine learning problem. It works through regression metrics (RMSE, MAE, MAPE, SMAPE, R squared, Pearson correlation, and RMSLE) and classification metrics (binary cross-entropy, accuracy, precision, recall, F1, and AUC), explaining the tradeoffs of each. Throughout, the framing is that metric choice is a business and risk question as much as a modeling one, using the same numeric error in different contexts to show why one metric can be right in one setting and wrong in another.

Key takeaways

Why no single metric works for every problem, and why optimizing for one metric can mean accepting worse results on others
The tradeoffs among regression metrics: RMSE penalizes large errors more heavily, MAE weights all errors equally, and MAPE and SMAPE express error as a percentage but struggle with zero or near-zero target values
How R squared and Pearson correlation let you compare model quality across different problems or industries by benchmarking against a simple mean-based prediction
Why RMSLE is useful for count and positive-value data, acting as a compressed, outlier-resistant version of RMSE
Why accuracy alone can be misleading on imbalanced problems, like predicting lottery winners, and how precision, recall, and F1 score fill in the picture
Why AUC is useful as a threshold-free way to compare how well a model ranks positive versus negative cases, illustrated with a debt-collection call-list example

Start owning your AI

One platform. Complete sovereignty. Measurable results.

Request a demo Talk to H2O

Overview