Model pruning

A trained neural network has many weights that barely matter: tiny numbers whose removal would hardly change any output. Pruning finds and removes them. Unstructured pruning zeroes out individual weights, which makes the model sparse. Structured pruning removes whole neurons, channels, attention heads or layers, which is easier for ordinary hardware to turn into real speed. Teams usually retrain briefly afterwards so the remaining weights compensate.

Business example. A company wants a speech model to run on a phone instead of in the cloud. Pruning away a large share of its weights can make it fit in memory and answer faster, and the remaining accuracy can be checked against the original model on the same test set before it ships.

Evidence. Han et al.’s 2015 paper cut the number of parameters in two well-known image models by 9 times and 13 times without losing ImageNet accuracy. The Lottery Ticket Hypothesis later argued that large networks contain much smaller subnetworks that could have been trained to similar accuracy on their own. Frameworks such as PyTorch include pruning utilities.

On the show. In episode 6, at 6:00, Galileo co-founder and CTO Atindriyo Sanyal says that if you host an open model yourself, “there’s various techniques that you can use like pruning and sparsification, to lower your inference time latencies.” Galileo sponsored Chain of Thought’s first season, and Conor was Galileo’s head of developer awareness when this episode was recorded, as he says in the episode intro.

Why it matters. Pruning is one of three common ways to make a model cheaper to run, with quantization, which stores weights in fewer bits, and knowledge distillation, which trains a smaller model to imitate a larger one. Editorial synthesis: sparse weights only save time if the serving software and chip can skip the zeros, so measured latency, not parameter count, is the number to check.