By Piyush Wairale · GATE DA Educator & Course Instructor, IIT Madras BS Programme · Updated August 2026

The Big Picture: One Linear Score, Three Different Jobs

All three models begin from the same object — a linear score w^Tx + b. What differs is what they do with it. Linear regression outputs the score directly as a continuous prediction. Logistic regression squashes it through a sigmoid into a probability for classification. Ridge regression keeps the linear-regression setup but penalises large weights so the model generalises better. Hold onto that framing and the “comparison” questions GATE loves become near-trivial.

Regression and classification are both named in the official GATE DA syllabus under Machine Learning — this post is the deep dive; the ML pillar guide maps the whole section.

Linear Regression: OLS Derived, Then Worked on Real Numbers

Model: ŷ = b₀ + b₁x. Ordinary least squares picks b₀, b₁ to minimise SSE = Σ(yᵢ − ŷᵢ)². Setting the two partial derivatives to zero gives the normal equations, whose solution every GATE aspirant should know cold:

The second identity means the fitted line always passes through (x̄, ȳ) — a fact GATE has tested directly.

Worked example (5 points)

Data: x = 1, 2, 3, 4, 5 and y = 2, 3, 5, 4, 6. Then x̄ = 3, ȳ = 4.

S_xy = (−2)(−2) + (−1)(−1) + 0(1) + 1(0) + 2(2) = 9.  S_xx = 4+1+0+1+4 = 10.

So b₁ = 9/10 = 0.9 and b₀ = 4 − 0.9(3) = 1.3, giving ŷ = 1.3 + 0.9x.

R² check: predictions are 2.2, 3.1, 4.0, 4.9, 5.8, so SSE = 0.04+0.01+1.00+0.81+0.04 = 1.9, while SST = Σ(yᵢ−ȳ)² = 10. Hence R² = 1 − 1.9/10 = 0.81 — and indeed r = S_xy/√(S_xxS_yy) = 9/√100 = 0.9, with r² = R² exactly as theory says for simple regression.

Multiple Regression and the Normal Equation

With p features, stack observations into X (n×(p+1), first column all 1s) and write ŷ = Xw. Minimising ‖y − Xw‖² gives the normal equation:

This closed form exists only when X^TX is invertible — i.e. columns of X are linearly independent. When features are correlated (multicollinearity), X^TX becomes near-singular, coefficient estimates blow up in variance, and you get wild, unstable weights. Remember that failure mode: it is precisely the disease ridge regression cures.

Logistic Regression: a Classifier Wearing a Regression Name

Logistic regression models the probability of the positive class: P(y=1|x) = σ(w^Tx + b), where the sigmoid σ(z) = 1/(1+e^−z) maps any real score into (0, 1). Equivalently, the log-odds are linear: log[p/(1−p)] = w^Tx + b. The decision boundary is where p = 0.5, i.e. w^Tx + b = 0 — a straight line/hyperplane, which is why logistic regression is a linear classifier.

Why not least squares? Squared error on probabilities gives a non-convex objective and poor gradients. Instead we maximise the likelihood of the data — equivalently minimise log-loss (binary cross-entropy): L = −Σ[ yᵢ log pᵢ + (1−yᵢ) log(1−pᵢ) ]. This is convex, but has no closed-form solution — training uses gradient descent or Newton’s method. That “closed form: no” cell is a favourite GATE MCQ.

Worked probability computation

Suppose a trained model has b = −4, w = 2 for a single feature: p(x) = σ(2x − 4).

At x = 2.5: z = 2(2.5) − 4 = 1, so p = 1/(1+e⁻¹) = 1/(1+0.368) = 0.731 → predict class 1.

At x = 2: z = 0 ⇒ p = 0.5 — x = 2 is exactly the decision boundary. And because log-odds are linear, each +1 in x multiplies the odds by e² ≈ 7.39. GATE regularly asks each of these three computations; knowing σ(0) = 0.5, σ(1) ≈ 0.731, σ(−1) ≈ 0.269 saves a minute.

Ridge Regression: L2 Shrinkage and the Bias–Variance Trade

Ridge modifies the OLS objective: minimise ‖y − Xw‖² + λ‖w‖². The penalty punishes large coefficients, and the closed form becomes:

Adding λI > 0 to X^TX makes it always invertible — ridge literally repairs the multicollinearity failure of the normal equation. Behaviourally: as λ increases, coefficients shrink smoothly toward (but never exactly to) zero, model variance falls, a little bias enters, and test error typically follows a U-shape. λ = 0 recovers OLS; λ → ∞ drives w → 0. Two practical notes GATE has tested: standardise features before applying the penalty, and the intercept is conventionally not penalised.

And lasso? Swap the penalty for L1 (λΣ|wᵢ|) and you get lasso, which can set coefficients exactly to zero — automatic feature selection, but no closed form. “Which penalty produces sparse weights?” is a recurring one-marker: L1 = sparse, L2 = shrunk-but-dense.

The Comparison Table GATE Questions Are Built From

Which Model When? A 10-Second Decision Rule

Ask two questions. Q1: Is the target continuous or a class label? Class label → logistic regression (or another classifier). Continuous → Q2. Q2: Are features many/correlated, or is the model overfitting? No → plain linear regression. Yes → ridge (keep all features, shrink them) or lasso (if you also want automatic feature selection). In exam MCQs, phrases like “probability of default”, “spam or not” ⇒ logistic; “highly correlated predictors”, “unstable coefficients” ⇒ ridge; “sparse model”, “feature selection” ⇒ lasso.

Three GATE-Style Problems, Solved

Problem 1 (NAT). For 10 points, Σx = 30, Σy = 50, Σxy = 200, Σx² = 140. Find the OLS slope and intercept.

Solution. x̄ = 3, ȳ = 5. b₁ = (Σxy − n x̄ȳ)/(Σx² − n x̄²) = (200 − 150)/(140 − 90) = 50/50 = 1.0. b₀ = 5 − 1(3) = 2.0. So ŷ = 2 + x.

Problem 2 (MCQ/NAT). A logistic model has b = 1, w = (2, −3). For x = (2, 1), find P(y=1) and the predicted class.

Solution. z = 1 + 2(2) − 3(1) = 2. p = 1/(1+e⁻²) = 1/(1+0.135) ≈ 0.881 → class 1. (Bonus: the odds are e² ≈ 7.39 to 1.)

Problem 3 (NAT). With standardised, orthonormal features, the OLS estimate of a coefficient is 3. What is its ridge estimate at λ = 0.5?

Solution. For orthonormal design X^TX = I, so w_ridge = (1+λ)⁻¹w_OLS = 3/1.5 = 2.0. This clean shrinkage formula — divide by (1+λ) — is the single most useful ridge fact for NAT questions.

Common Mistakes to Avoid

Calling logistic regression a regression model. The output is a probability and the task is classification; “logistic regression minimises squared error” is a standard false statement in MCQs.

Confusing R² with r. In simple linear regression R² = r²; an r of −0.9 still gives R² = 0.81. Watch the sign trap.

Claiming ridge performs feature selection. Ridge shrinks weights toward zero but never exactly to zero — that’s lasso (L1).

Forgetting the line passes through (x̄, ȳ). Many NAT questions are solvable in one step from this fact alone.

Using accuracy formulas on z instead of σ(z). First compute the score, then squash. σ(2) ≈ 0.881, not 2.

How GATE DA Asks Regression

Expect: (1) NAT — compute slope/intercept from raw data or summary sums, or a predicted ŷ at a new x; (2) NAT — sigmoid probability from given weights, or an odds ratio; (3) MCQ — properties: convexity, closed forms, what λ does to bias/variance, L1 vs L2 sparsity; (4) linked — regression inside a hypothesis-testing question (significance of a slope) or combined with the probability & statistics toolkit via MLE. The arithmetic is always small — the marks are for knowing which formula applies.

FAQs: Regression for GATE DA

Keep building: revise the complete ML roadmap, strengthen the statistics behind it with the Probability & Statistics pillar, and check the GATE DA 2027 syllabus to track your coverage. New ML problem-solving sessions drop regularly on my YouTube channel — subscribe so you don’t miss them.