Neural architecture search

Designing a neural network involves many choices: how many hidden layers, how wide each one is, which operations to use and how layers connect. Engineers usually make those choices by experience and trial and error. Neural architecture search hands the job to an algorithm. It proposes candidate designs, trains or estimates how well each would perform, and uses the results to propose better ones.

Business example. A phone maker wants an image model that runs within a strict battery and latency budget on its own chip. NAS can search for the most accurate design that still meets that budget, a trade-off that is hard to tune by hand.

How it works. Elsken, Metzen and Hutter’s survey breaks every NAS method into three parts. The search space defines which designs are allowed. The search strategy, which can be reinforcement learning, evolution or gradient-based methods, decides what to try next. The performance-estimation strategy decides how to score a candidate without fully training every one. Zoph and Le’s 2016 paper popularized the field by training a controller network with reinforcement learning to write architectures.

Cost. Early NAS was extremely expensive because each candidate had to be trained. Much later work focused on cheaper estimates, such as sharing weights between candidates. NAS sits within the wider field of automated machine learning, together with hyperparameter optimization and meta-learning, as the AutoML book lays out.

Why it matters. NAS found some strong image architectures and is still used to fit models to specific hardware. Editorial synthesis: for frontier large language models, most gains have come from scaling a fixed transformer design with more data and compute, so labs tend to tune a few architecture choices rather than search for whole new designs.