By Piyush Wairale · GATE DA Educator & Course Instructor, IIT Madras BS Programme · Updated August 2026

From Perceptron to Multi-Layer Perceptron

A single perceptron computes a weighted sum plus bias and applies an activation: output = f(w·x + b). It can only draw one hyperplane, so it solves linearly separable problems and nothing else. Stack perceptrons into layers — every unit connected to every unit in the next layer — and you get the multi-layer perceptron (MLP): input layer (no parameters, just the features), one or more hidden layers, and an output layer. Feedforward networks are named in the official GATE DA syllabus, and the exam’s favourite question about them is disarmingly concrete: how many trainable parameters does this architecture have? The ML pillar guide shows where networks sit in the wider preparation order.

Parameter Counting: One Formula, Worked Twice

Worked 1 — architecture 4–5–3: Input→hidden: 4×5 + 5 = 25. Hidden→output: 5×3 + 3 = 18. Total = 25 + 18 = 43 trainable parameters. The input layer contributes nothing — it holds data, not weights.

Worked 2 — architecture 6–8–4–2 (two hidden layers): 6×8 + 8 = 56; 8×4 + 4 = 36; 4×2 + 2 = 10. Total = 56 + 36 + 10 = 102. In the diagram above, 3–4–2 gives 16 + 10 = 26 — count it yourself as a drill. Variants GATE plays: “without biases” (drop the +n_out terms: 4–5–3 → 35), “weights only in the second layer” (just 5×3 = 15), or “how many connections?” (same as weight count). Read which quantity is asked before multiplying.

Activation Functions: the Table to Memorise

Why Nonlinearity Is Non-Negotiable: XOR

Compose two linear layers with no activation: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁+b₂) — still one linear map. Depth buys nothing without nonlinearity. The classic witness is XOR: points (0,0), (1,1) in one class and (0,1), (1,0) in the other cannot be split by any single line (Minsky & Papert’s perceptron limitation), but a 2-unit hidden layer with a nonlinear activation carves the plane into the two diagonal regions easily. “Minimum hidden units to solve XOR?” — 2 — is a recurring one-marker, the same separability story told for kernels in the SVM guide.

A Forward Pass, Computed by Hand

Network 2–2–1, all sigmoid. Input x = (1, 0). Hidden unit 1: weights (0.5, 0.5), bias 0. Hidden unit 2: weights (1, −1), bias 0. Output unit: weights (1, 1), bias −1.

Hidden layer: z₁ = 0.5(1) + 0.5(0) = 0.5 → h₁ = σ(0.5) = 0.622. z₂ = 1(1) − 1(0) = 1 → h₂ = σ(1) = 0.731.

Output layer: z_out = 1(0.622) + 1(0.731) − 1 = 0.353 → ŷ = σ(0.353) ≈ 0.587.

Every GATE forward-pass NAT is this pattern: weighted sum, add bias, apply the named activation, repeat layer by layer. Keep σ(0.5) ≈ 0.622, σ(1) ≈ 0.731, σ(2) ≈ 0.881 handy — the same sigmoid anchors from the logistic regression guide.

Backpropagation and One Step of Gradient Descent

Backpropagation is the chain rule, organised. The loss depends on the output, the output on the last layer’s weights and inputs, those inputs on the previous layer, and so on — so ∂L/∂w for any weight is a product of local derivatives along the path from that weight to the loss. Computing these products backward layer by layer reuses shared factors, which is why one backward pass costs about as much as one forward pass.

Vanishing gradients: with sigmoid activations every chain factor includes σ′ ≤ 0.25, so a 10-layer chain shrinks gradients by up to 0.25¹⁰ ≈ 10⁻⁶ — early layers barely learn. This is the standard MCQ answer for “why ReLU in deep networks?”: its derivative is 1 for positive inputs, so gradients survive depth.

Gradient descent, one worked step: take L(w) = (w − 3)², w₀ = 0, learning rate η = 0.25. Gradient: ∇L = 2(w−3) = −6 at w₀. Update: w₁ = 0 − 0.25(−6) = 1.5. Next step: ∇L = 2(1.5−3) = −3, so w₂ = 1.5 + 0.75 = 2.25 — halving the gap to the optimum each step. Too large an η overshoots and diverges; too small crawls — the tuning trade-off connects directly to the bias–variance and validation machinery.

Softmax + Cross-Entropy, Computed

Output logits z = (2, 1, 0). Exponentials: e² = 7.389, e¹ = 2.718, e⁰ = 1. Sum = 11.107. Softmax probabilities: p = (7.389, 2.718, 1)/11.107 = (0.665, 0.245, 0.090) — they sum to 1, and the largest logit gets the largest probability (softmax preserves ranking; subtracting a constant from every logit changes nothing).

If the true class is the first, cross-entropy loss = −ln(0.665) ≈ 0.408. And the elegant fact that makes deep learning trainable: with softmax + cross-entropy, the output-layer gradient is simply p − y = (0.665−1, 0.245, 0.090) = (−0.335, 0.245, 0.090).

Three GATE-Style Problems, Solved

Problem 1 (NAT). How many trainable parameters (with biases) in a 10–20–15–5 MLP?

Solution. 10×20+20 = 220; 20×15+15 = 315; 15×5+5 = 80. Total = 615.

Problem 2 (NAT). Network 2–2–1 with ReLU hidden units, linear output. x = (2, −1). Hidden: h₁ weights (1, 2), bias −1; h₂ weights (0, 1), bias 2. Output weights (3, 2), bias 0. Find the output.

Solution. z₁ = 2 − 2 − 1 = −1 → ReLU → 0. z₂ = −1 + 2 = 1 → 1. Output = 3(0) + 2(1) + 0 = 2. Note how ReLU killing h₁ simplified everything — watch for that design in exam questions.

Problem 3 (MCQ). Which is TRUE? (a) sigmoid outputs can be negative; (b) tanh outputs lie in (−1, 1); (c) ReLU’s derivative at z = −2 is 1; (d) softmax outputs can exceed 1.

Solution. (b). Sigmoid is strictly positive; ReLU’s derivative is 0 for negative inputs (the “dying ReLU” region); softmax entries are probabilities in (0, 1).

Common Mistakes to Avoid

Forgetting biases — or including them when the question says “weights only”. The two answers differ by exactly the total number of non-input units.

Counting the input layer as parameterised. It isn’t; parameters live on connections into a layer.

Applying the activation before adding the bias. Order is: weighted sum → + bias → activation.

Sign slips in the update rule. w ← w minus η∇L — you descend the gradient, not climb it.

Normalising softmax with a sum of logits instead of exponentials. Exponentiate first, then divide by the sum of exponentials.

How GATE DA Asks Neural Networks

Recurring shapes: (1) NAT — parameter/connection counting for a stated architecture (the most predictable 1–2 marks in the paper); (2) NAT — a small forward pass, often with ReLU zeroing one unit, or one gradient-descent step; (3) NAT — softmax probabilities or a cross-entropy value; (4) MCQ — activation ranges/derivatives, vanishing gradients, why nonlinearity/XOR. Arithmetic stays small; layout discipline wins the marks.

FAQs: Neural Networks for GATE DA

Round out your prep: revisit regularisation and evaluation in the bias–variance & cross-validation guide, connect the sigmoid story back to logistic regression, and track your coverage against the GATE DA 2027 syllabus. New ML problem-solving sessions drop regularly on my YouTube channel — subscribe so you don’t miss them.