Every neural network is a large set of numbers. When a model is trained from scratch, most of them start as random values; training adjusts them until the model’s predictions are good. Those numbers are the parameters, and in a large language model there are billions of them. When a company releases open weights, the parameters are what it publishes.
Parameter count is a rough guide to a model’s capability and cost. More parameters can store more knowledge and patterns, and among models of similar design they generally take more memory and computing power to run, which raises the cost and latency of each answer. Design choices change the math: a mixture-of-experts model uses only some of its parameters for each token, and quantization shrinks the memory each one takes. Size isn’t everything: on the prompts OpenAI’s InstructGPT research tested, people preferred a 1.3-billion-parameter model trained to follow instructions over the 175-billion-parameter GPT-3. That is why teams often move stable, narrow tasks to a small language model.
Two parameters you can see
A tiny model makes the idea concrete. Suppose a trained linear model predicts y = 2x + 1. The weight 2 and the bias 1 are its two parameters. The input x is not: at x = 3 the prediction is 7, at x = 5 it is 11, and using the model on those inputs changes neither number. A large language model works the same way with billions of values, which jointly encode patterns rather than storing one fact per number.
Three things are easy to confuse with parameters. A hyperparameter, such as the learning rate or batch size, is a setting that controls training, not a value the model learns (Google, hyperparameters). A prompt is input: changing it changes what the model reads, not its weights; fine-tuning is what changes weights. And storage depends on precision as well as count: two parameters take 8 bytes at 32 bits each and 4 bytes at 16 bits, before metadata, and running a model also needs memory for things like the KV cache, so weight size alone is not a hardware plan.
Sources
- Ouyang et al.: Training language models to follow instructions with human feedback — Reports that on the authors' prompt distribution, human raters preferred outputs from the 1.3B-parameter InstructGPT model to those of the 175B-parameter GPT-3; checked October 26, 2026.
- Google Machine Learning Crash Course: Hyperparameters — Hyperparameters such as learning rate, batch size and epochs control training; parameters, like the weights and bias, are part of the model itself. Checked November 6, 2026.
Go deeper
- How do large language models actually work? AI, decoded · How Large Language Models Actually Work
- When should you use a small language model instead of a frontier model in production? AI, decoded · Small Language Model vs. Frontier Model
- Are small language models better than large ones for production? AI, decoded · Small Language Models in Production