Hyperparameter
A hyperparameter is a setting chosen outside the fitting of a model's learned parameters, such as learning rate, batch size or tree depth. Validation results can guide its selection by a person or an automated search.
For a classifier built from decision trees, a team can try several maximum depths. Training learns the splits inside each tree; maximum depth limits the structure the training procedure can build. Choose among the candidate settings using validation data, then assess the chosen system on a separate test set.
Hyperparameters can describe architecture, optimization or regularization. They need not stay constant throughout training: a learning-rate schedule can vary the rate over time, while the schedule itself is chosen outside ordinary parameter fitting. Automated search changes how settings are selected, not their distinction from learned weights.
This matters because repeatedly choosing settings from test-set results turns that set into another tuning resource. Its score then becomes a less trustworthy estimate of performance on unseen work. Record the settings and selection procedure so comparisons can be reproduced. Generation controls, such as temperature, are sometimes called inference hyperparameters; they affect decoding rather than fitting the model.
Sources
- scikit-learn: Tuning hyperparameters — Distinguishes settings from directly learned parameters and describes validation-based search.
Go deeper
- Keras: Getting started with KerasTuner docs
Define a search space and tune neural-network settings against validation results.
