Skip to content

AI Glossary · Concepts

What is Model Parameters?

Model Parameters

Parameters are the learned numbers inside an AI model, also called weights, that training adjusts and that together store what the model has learned. The parameter count, such as 70 billion, is the usual shorthand for a model's size.

· Updated · Chain of Thought

Model Architecture

  • 3 explainers
  • 4 episodes
  • 2 sources

Every neural network is a large set of numbers. When a model is trained from scratch, most of them start as random values; training adjusts them until the model’s predictions are good. Those numbers are the parameters, and in a large language model there are billions of them. When a company releases open weights, the parameters are what it publishes.

Parameter count is a rough guide to a model’s capability and cost. More parameters can store more knowledge and patterns, and among models of similar design they generally take more memory and computing power to run, which raises the cost and latency of each answer. Design choices change the math: a mixture-of-experts model uses only some of its parameters for each token, and quantization shrinks the memory each one takes. Size isn’t everything: on the prompts OpenAI’s InstructGPT research tested, people preferred a 1.3-billion-parameter model trained to follow instructions over the 175-billion-parameter GPT-3. That is why teams often move stable, narrow tasks to a small language model.

Two parameters you can see

A tiny model makes the idea concrete. Suppose a trained linear model predicts y = 2x + 1. The weight 2 and the bias 1 are its two parameters. The input x is not: at x = 3 the prediction is 7, at x = 5 it is 11, and using the model on those inputs changes neither number. A large language model works the same way with billions of values, which jointly encode patterns rather than storing one fact per number.

Three things are easy to confuse with parameters. A hyperparameter, such as the learning rate or batch size, is a setting that controls training, not a value the model learns (Google, hyperparameters). A prompt is input: changing it changes what the model reads, not its weights; fine-tuning is what changes weights. And storage depends on precision as well as count: two parameters take 8 bytes at 32 bits each and 4 bytes at 16 bits, before metadata, and running a model also needs memory for things like the KV cache, so weight size alone is not a hardware plan.

Sources

Go deeper

From the conversation