Bias-Variance Tradeoff
The fundamental tension in ML between a model that is too simple (high bias, underfitting) and one that is too complex (high variance, overfitting). The goal is finding the sweet spot.
Why It Matters
Understanding the bias-variance tradeoff is essential for model selection and debugging. It explains why adding complexity does not always improve results.
Example
A linear model (high bias) that consistently predicts house prices $50K too low versus a complex model (high variance) that is sometimes $100K off in either direction.
Think of it like...
Like a golf swing — too rigid (high bias) and you consistently miss in one direction, too loose (high variance) and your shots scatter everywhere. The sweet spot is controlled flexibility.
Related Terms
Overfitting
When a model learns the training data too well — including its noise and random fluctuations — and performs poorly on new, unseen data. The model essentially memorizes rather than generalizes.
Underfitting
When a model is too simple to capture the underlying patterns in the data, resulting in poor performance on both training and new data. The model has not learned enough from the training data.
Regularization
Techniques used to prevent overfitting by adding constraints or penalties to the model during training. Regularization discourages the model from becoming too complex or fitting noise in the training data.
Generalization
A model's ability to perform well on new, unseen data that was not part of its training set. Generalization is the ultimate goal of machine learning — learning patterns, not memorizing examples.