Parameter
Any learnable value in a machine learning model that is adjusted during training. Parameters include weights and biases in neural networks. Model size is often described by parameter count.
Why It Matters
Parameter count is a rough proxy for model capability — 7B, 70B, and 1.7T parameter models represent very different capability tiers.
Example
GPT-4 is estimated to have over 1 trillion parameters, while Llama 2 comes in 7B, 13B, and 70B parameter versions.
Think of it like...
Like the number of adjustable knobs in a recording studio — more knobs give finer control, but also require more expertise and resources to tune properly.
Related Terms
Weight
A numerical parameter in a neural network that is learned during training. Weights determine the strength of connections between neurons and collectively encode the model's knowledge.
Hyperparameter
Settings that are configured before training begins and control how the model learns, as opposed to parameters which are learned during training. Examples include learning rate, batch size, and number of layers.
Model Size
The number of parameters in a model, typically expressed in millions (M) or billions (B). Model size correlates loosely with capability but also determines compute and memory requirements.
Scaling Laws
Empirical findings showing predictable relationships between model performance and factors like model size (parameters), dataset size, and compute budget. Performance improves as a power law with these factors.
Neural Network
A computing system inspired by the biological neural networks in the human brain. It consists of interconnected nodes (neurons) organized in layers that process information and learn to recognize patterns.