Regularization
Regularization changes model training to discourage solutions that fit training details too closely and fail on new data. It includes penalties and constraints, as well as techniques such as dropout and early stopping.
A price-prediction model can use an L2 penalty that adds a cost for large coefficients to its training objective. The model must balance fitting the observed prices against that penalty. This favors smaller coefficients, rather than chasing every training fluctuation.
Regularization is broader than weight penalties. Dropout randomly omits selected activations during training. Early stopping ends training when a chosen validation measure stops improving and can retain the earlier model. These methods intervene differently; they are not interchangeable settings.
The benefit must be measured on held-out examples. Too little constraint can allow overfitting; too much can prevent the model from learning useful patterns. The regularization strength is a hyperparameter to select for the task. Neither a penalty nor early stopping repairs incorrect labels, leaked test data or an unrepresentative evaluation.
Sources
- Google: L2 regularization — Explains an objective combining training loss with a penalty on squared weights.
- Keras: Training and evaluation with built-in methods — Shows dropout and an early-stopping callback monitoring validation loss.
Go deeper
- Keras: Dropout layer docs
Inspect a regularizer that masks activations during training rather than penalizing weight size.