- Model parameters, such as neural network weights and biases, are learned during training; hyperparameters are set by the developer before training.
- Hyperparameter optimization is the process of finding the best set of hyperparameter values for a given data set and problem.
- The search space treats each hyperparameter as a dimension, and each point in it is one possible model configuration.
- Common search methods are manual search, grid search, random search and Bayesian optimization.
- Good hyperparameters help a model generalize, performing as well on new data as on training data.
Two types of parameters govern the operation of machine learning (ML) models, model parameters and hyperparameters. Model parameters are parameters the model has learned during training, such as weights and biases in a neural network. Hyperparameters are settings and regulations that are set by the developer before training to control how the algorithm applies the model to the data. When a model is first being created, the best combinations and settings for the hyperparameters are unknown. This creates a need for hyperparameter optimization.
Hyperparameter optimization is the process of finding the best set of hyperparameter values for a given data set and problem. It begins by identifying the search space, which is a theoretical space where each hyperparameter to be optimized acts as a dimension; each point in this space represents a possible configuration of the model. There are several methods developers use to search through these configurations and identify the best candidates.
Hyperparameter Optimization Methods
Manual Search
Manual search is the traditional method of hyperparameter optimization. Users manually select several settings for each hyperparameter and then compare the performance of the model with each combination of those selected settings. For instance, users might select ntrees settings of 50, 100, and 200, and max_depth settings of 5, 10, 15, and 20. This would give users twelve combinations and twelve models to compare. However, this method becomes less practical when there are many hyperparameters and settings to compare.
Grid Search
A grid search is similar to a manual search. It involves determining a set of values for each hyperparameter and evaluating the model with each possible combination of these values. It is less tedious than manual search but still involves the developers guessing at which values may be optimal.
Random Search
The random search method involves the system randomly selecting the values for each hyperparameter and then running the model to see how it performs. Random search often yields models equal to or better than grid and manual search in less time. Comparing the models it generates lets developers narrow the search to the areas that look promising.
Bayesian Optimization
Bayesian optimization involves running sets of hyperparameters then using the results to improve the hyperparameter sets for the next search. Bayesian optimization is more efficient than other methods and can reduce optimization time and improve results.
Why Does Hyperparameter Optimization Matter?
Hyperparameter optimization is essential to machine learning and AI systems because it allows developers to identify the best configurations of a model for a given situation. It helps machine learning models generalize, which is the ability to operate just as effectively whether encountering training data or new data. This leads to better performance of the model and better results for users.
Hyperparameter Optimization in H2O AI
Open-source H2O-3 supports grid search and random search over hyperparameters and provides default values as a starting point. Its Flow interface groups hyperparameters by importance and marks the critical ones. H2O-3 grid search documentation.