A team predicting which customers will cancel might try 30 combinations of tree depth, learning rate and number of trees, train a model for each, and keep the combination with the best score on validation data. The settings are hyperparameters: chosen around training rather than learned during it. Each trial costs a full or partial training run, so the search strategy matters.
Grid search tries every combination on a fixed grid, which wastes trials when only one or two settings really matter. James Bergstra and Yoshua Bengio found that random search, which samples combinations at random, found equally good or better models in a small fraction of the computation. Bayesian optimization goes further and uses earlier results to pick the next trial. Tools often add early stopping to abandon unpromising trials. Searching over model types and data preparation as well is AutoML.
Keeping track of trials is half the job. In episode 22 at 3:07, Databricks’ Denny Lee describes losing track of which version of the hyperparameters went with which model, and which dataset was used, the problem experiment-tracking tools exist to solve. Galileo sponsored Season 2, including episode 22; Conor notes Databricks’ investment in Galileo at 13:44. In episode 17 at 11:23, IBM’s Maryam Ashoori names hyperparameter optimization as one angle of cost-performance work that needs AI skills, and at 11:48 describes IBM automating comparisons of retrieval pipelines with different parameters. Galileo also sponsored episode 17; co-host Atindriyo Sanyal, Galileo’s co-founder and CTO, is introduced at 0:20. Finally, tuning against one validation set can overfit to it, so keep a separate test set for the final check.
Hear it from the guest
“And then you forget which set of hyperparameters were associated to which model.”
“Hyperparameter optimization is one angle.”
Quotes lightly edited to remove filler words.
Sources
- scikit-learn: Tuning the hyper-parameters of an estimator — Describes searching a hyperparameter space with exhaustive grid search and randomized search, scored by cross-validation; checked October 24, 2026.
- Bergstra and Bengio: Random Search for Hyper-Parameter Optimization (JMLR, 2012) — Finds that random search over the same domain finds models as good as or better than grid search within a small fraction of the computation time; checked October 24, 2026.
- Snoek, Larochelle and Adams: Practical Bayesian Optimization of Machine Learning Algorithms — Applies Bayesian optimization to choose each next hyperparameter trial based on the results of earlier trials; checked October 24, 2026.
