- A validation set is used during training to tune and optimize a model, separate from the training and test sets.
- The main purpose of a validation set is to prevent overfitting.
- A common split is roughly 60% training, 20% validation and 20% testing.
- Data should be shuffled before it is split into sets.
- Models with more hyperparameters need a larger validation set.
The primary purpose of a validation set is to prevent the model from overfitting. Overfitting is when a model memorizes a pattern in the training data, but cannot make accurate predictions on what it has not yet reviewed. As a result, the model will review the data, but not learn from it.
Training Vs. Testing Vs. Validation Sets
Understanding the difference between the datasets for training and testing the model, and how to split the dataset, is essential to machine learning.
Training Set
A training set is the data used in training an AI model. The model makes correct decisions based on the data it sees and learns from. Training data is taken from several resources and then organized to ensure proper performance of the model. On average 60% of collected data is used for the training set.
Validation Set
A validation set is used to fine tune and optimize a model, and is considered to be part of the training phase. Applying the validation set helps prevent overfitting. Roughly 20% of the collected data is used in the validation set.
Testing Set
Test datasets are the benchmark for evaluating a model. The data used in a testing set is taken from real-world situations the model will likely face. The testing set uses about 20% of the data collected, and comes after the training and validation of the model are finished.
Dataset Split Ratio
When splitting a dataset, there is no perfect ratio. Distributing this data properly ensures that the model learns rules from the given data. The primary guideline is to allocate the majority of collected data to the training set. Additionally, before splitting the data, it must be shuffled
The ideal dataset split ratio depends on the number of samples present in the dataset and the model. The more hyperparameters there are, the larger the validation set needs to be to optimize the model. However, if there are few to no hyperparameters, a smaller dataset can be used to validate the model.