- Target leakage happens when information that would not be known at prediction time is used to train a model.
- Leakage makes a model look more accurate in training than it will be on real, unseen data.
- Normalizing or imputing with statistics from the whole training dataset is one way leakage creeps into complex datasets.
- A variable updated after the outcome is known, like taking antibiotics after a sinus infection, leaks the target.
- Avoid target leakage by omitting data that is not known at the time of the target outcome.
What is Target Leakage?
Target leakage can happen when a variable that is not a feature is being used to predict the target.
Target leakage happens more frequently with complex datasets. For instance, if the training dataset was normalized or standardized by using missing value imputation (min, max, mean). An unseen dataset may not have any knowledge of the distribution of data in the dataset. For example, a row in the unseen dataset would not know if there was one record or one million other records. The dataset would not be able to normalize or standardize itself using unseen data. This results in the model overfitting to the training data and produces a higher accuracy when run on unseen data than it would have if the model had not fit the training dataset.
Target Leakage Examples
Try to think about target leakage occurring over time when new data becomes available and changes existing data used in the model.
Consider the following example. You want to predict who will get sick with a sinus infection. Your data includes two columns: got_sinus_infection and took_antibotic.
Most people tend to take antibiotics after getting a sinus infection so they can recover. This raw data shows a strong relationship between got_sinus_infection and took_antibotic, however, took_antibotic is commonly changed after the value for got_sinus_infection has been determined. This results in target leakage.
Why is Target Leakage Important?
Target leakage is a consistent problem in data science and machine learning. It can cause a model to underestimate its generalization error, making it useless for any real-world application.
Target leakage is particularly challenging because it can be either intentional or unintentional, making it difficult to pinpoint. Kaggle contestants have been known to intentionally include sampling errors that result in target leakage as a way to develop accurate models and gain competitive advantages in data science competitions.
Target Leakage in H2O AI
H2O Driverless AI, our AutoML, checks for leakage: it builds a simple model on each important feature and reports features with unusually high AUC or R2 as potential leaks. Leakage detection in Driverless AI.