Every neural network is a large set of numbers. Before training they are random; training adjusts them until the model’s predictions are good. Those numbers are the parameters, and in a large language model there are billions of them. When a company releases open weights, the parameters are what it publishes.
Parameter count is a rough guide to a model’s capability and cost. More parameters can store more knowledge and patterns, and among models of similar design they generally take more memory and computing power to run, which raises the cost and latency of each answer. Design choices change the math: a mixture-of-experts model uses only some of its parameters for each token, and quantization shrinks the memory each one takes. Size isn’t everything: on the prompts OpenAI’s InstructGPT research tested, people preferred a 1.3-billion-parameter model trained to follow instructions over the 175-billion-parameter GPT-3. That is why teams often move stable, narrow tasks to a small language model.
