By Piyush Wairale · GATE DA Educator & Course Instructor, IIT Madras BS Programme · Updated August 2026

The Error Decomposition: Where Test Error Comes From

Fix a test point x and imagine retraining your model on many different random training sets. Each retraining gives a slightly different prediction f̂(x). The expected squared error against the true value decomposes as:

Bias is the systematic gap between the average prediction and the truth — the model family is too simple to capture the pattern. Variance is how much predictions wobble across training sets — the model is so flexible it memorises the accidents of each sample. Noise σ² is randomness in y itself; no model can remove it, which is why 100% test accuracy claims should make you suspicious. This decomposition is the single most connective idea in the GATE DA syllabus — it explains pruning, regularisation, k in kNN and C in SVM with one picture.

Underfitting vs Overfitting: Diagnose From Two Numbers

The U-Curve Every GATE Aspirant Must Be Able to Sketch

Training error only ever falls as complexity grows — a rich enough model can memorise anything. Test error falls while added complexity captures real signal, bottoms out at the sweet spot, then rises as the model starts fitting noise. Left of the minimum you are bias-dominated; right of it, variance-dominated. GATE questions describe a scenario and ask which zone you’re in — the two-error table above answers it mechanically.

Every Hyperparameter Is the Same Dial

GATE never asks the decomposition in isolation — it asks how a specific knob moves you along the curve. The four that matter:

Tree depth (see decision trees): deeper → lower bias, higher variance. Pruning walks you back toward the sweet spot.

k in kNN: k = 1 memorises the training set (low bias, high variance); large k averages over many neighbours (high bias, low variance). Note the direction: increasing k increases bias.

λ in ridge (see the regression comparison): larger λ shrinks weights → bias up, variance down.

C in SVM (see the SVM guide): larger C punishes violations → bias down, variance up — the opposite direction to λ, because C multiplies the error term (C ≈ 1/λ).

Memorise those four directions as one sentence: depth↑ and C↑ move right on the U-curve; k↑ and λ↑ move left.

Learning Curves: What More Data Buys You

Plot error against training-set size instead of complexity. Training error rises with more data (harder to memorise), validation error falls, and the two converge. Read the gap: curves converging to a high plateau = high bias — more data won’t help, get a richer model. A large persistent gap between low train and high validation error = high variance — more data (or regularisation) will help. “Will collecting more data improve this model?” is a GATE scenario question answered entirely by which pattern you’re shown.

Cross-Validation: Estimating Test Error Honestly

A single train/test split (holdout) wastes data and gives a noisy estimate. k-fold cross-validation fixes both: partition the data into k equal folds; for each fold in turn, train on the other k−1 folds and validate on it; average the k scores.

Worked: 5-fold CV on 100 samples

Folds of 100/5 = 20 samples each. Each run trains on 80, validates on 20. Total model fits: 5. If the five validation accuracies are 0.85, 0.80, 0.90, 0.82, 0.88, the CV estimate is their mean: (0.85+0.80+0.90+0.82+0.88)/5 = 0.85. Every sample was validated on exactly once and trained on exactly 4 times.

LOOCV is k-fold with k = n: n fits of n−1 samples each — nearly unbiased but expensive (n = 1,000 means 1,000 trainings) and high-variance as an estimator. Stratified k-fold preserves class proportions inside every fold — essential for imbalanced data (a 95:5 dataset can otherwise produce folds with zero minority samples).

Choosing k, and the Data-Leakage Trap

Choosing k is itself a bias–variance decision. Small k (e.g. 2–3): each model trains on little data → pessimistic (biased) error estimates, but cheap. Large k (up to n): training sets are nearly full-size → low bias in the estimate, but expensive and the k estimates are highly correlated (variance). k = 5 or 10 is the standard compromise — the answer to “why 10-fold?” MCQs.

Data leakage: any preprocessing that looks at the whole dataset — scaling to global mean/variance, feature selection on all rows, imputation from global statistics — must be fitted inside each training fold only, then applied to that run’s validation fold. Preprocess-then-split lets validation information leak into training and inflates CV scores. “Standardised the full dataset before 10-fold CV — what’s wrong?” has appeared in GATE-adjacent papers; the answer is leakage, and the estimate is optimistically biased.

Three GATE-Style Problems, Solved

Problem 1 (MCQ). Model A: train error 2%, test error 18%. Model B: train error 15%, test error 16%. Diagnose each.

Solution. A has a huge train–test gap → overfitting (high variance): regularise, prune or add data. B has both errors high and close → underfitting (high bias): increase capacity or reduce regularisation. Note B actually has the better test error — “lower training error” never by itself means the better model.

Problem 2 (NAT). You tune 4 candidate values of λ using 5-fold CV, then retrain the winner on all data. How many model fits in total?

Solution. Each λ costs 5 fits → 4 × 5 = 20, plus 1 final refit = 21. (Full nested CV with an outer 5-fold loop would cost 5 × (4×5 + 1) = 105 — count carefully what the question includes.)

Problem 3 (MCQ). In kNN, k is increased from 1 to 25. What happens to bias and variance?

Solution. Bias increases, variance decreases. k = 1 fits every quirk of the sample; k = 25 averages over a broad neighbourhood, smoothing the boundary. Training error at k = 1 is 0 by construction — another reminder that training error is not a quality metric.

Common Mistakes to Avoid

Treating training error as the target. It’s a diagnostic. Model selection uses validation/CV error, final reporting uses a untouched test set.

Reversing the C direction. C behaves like 1/λ: large C = less regularisation = overfitting risk. Check against the SVM table before answering.

Forgetting irreducible noise. Bias² + variance alone understates expected error; the σ² term is why the U-curve’s minimum sits above zero.

Counting CV fits wrong. k-fold on m hyperparameter candidates = m×k fits (+1 refit if the question says so). Write the multiplication out.

Leaky preprocessing. Fit scalers, encoders and feature selectors on training folds only.

How GATE DA Asks This Topic

Recurring forms: (1) MCQ — diagnose over/underfitting from train/test numbers or a described learning curve; (2) MCQ — direction questions: what happens to bias/variance when depth, k, λ or C changes; (3) NAT — count folds, fold sizes or model fits; (4) MCQ — properties of LOOCV vs k-fold vs stratified, or spot the leakage. It’s the connective tissue of the ML section — revise it right after each classifier in the ML pillar roadmap.

FAQs: Bias–Variance & Cross-Validation for GATE DA

Connect the dots: see the dial in action with λ in ridge regression, C in SVM and pruning in decision trees, then track your coverage against the GATE DA 2027 syllabus. New ML problem-solving sessions drop regularly on my YouTube channel — subscribe so you don’t miss them.