Neural Network Parameter Counting for GATE DA: MLPs, Activations & Worked Examples
The most predictable NAT in the paper — parameter counting worked on two full examples — plus a forward pass computed by hand, every activation function you must know, and backprop without the tears.
By Piyush Wairale · GATE DA Educator & Course Instructor, IIT Madras BS Programme · Updated August 2026
Key Takeaways
• A dense layer from nin to nout units has nin×nout weights + nout biases. Sum over layers — a 4–5–3 MLP has 25 + 18 = 43 parameters.
• Without a nonlinear activation, any stack of linear layers collapses into one linear map — that’s why XOR needs a hidden layer and a nonlinearity.
• Know four activations cold: sigmoid (0,1), tanh (−1,1), ReLU [0,∞), softmax (probability vector). Sigmoid’s derivative σ(1−σ) ≤ 0.25 explains vanishing gradients.
• Training = forward pass → loss → backprop gradients by the chain rule → update w ← w − η∇L. GATE asks each stage as a small numeric.
On this page
Perceptron to MLP · Parameter counting (worked twice) · Activation functions · Why nonlinearity (XOR) · Forward pass on numbers · Backprop & gradient descent · Softmax & cross-entropy · Solved problems · Mistakes · Exam patterns · FAQs
From Perceptron to Multi-Layer Perceptron
A single perceptron computes a weighted sum plus bias and applies an activation: output = f(w·x + b). It can only draw one hyperplane, so it solves linearly separable problems and nothing else. Stack perceptrons into layers — every unit connected to every unit in the next layer — and you get the multi-layer perceptron (MLP): input layer (no parameters, just the features), one or more hidden layers, and an output layer. Feedforward networks are named in the official GATE DA syllabus, and the exam’s favourite question about them is disarmingly concrete: how many trainable parameters does this architecture have? The ML pillar guide shows where networks sit in the wider preparation order.
Parameter Counting: One Formula, Worked Twice
Worked 1 — architecture 4–5–3: Input→hidden: 4×5 + 5 = 25. Hidden→output: 5×3 + 3 = 18. Total = 25 + 18 = 43 trainable parameters. The input layer contributes nothing — it holds data, not weights.
Worked 2 — architecture 6–8–4–2 (two hidden layers): 6×8 + 8 = 56; 8×4 + 4 = 36; 4×2 + 2 = 10. Total = 56 + 36 + 10 = 102. In the diagram above, 3–4–2 gives 16 + 10 = 26 — count it yourself as a drill. Variants GATE plays: “without biases” (drop the +nout terms: 4–5–3 → 35), “weights only in the second layer” (just 5×3 = 15), or “how many connections?” (same as weight count). Read which quantity is asked before multiplying.
▶ Watch: my complete Machine Learning playlist for GATE DA
Neural network numericals solved on camera — all playlists →
Activation Functions: the Table to Memorise
| Function | Formula | Range | Derivative |
|---|---|---|---|
| Sigmoid | 1/(1+e−z) | (0, 1) | σ(1−σ), max 0.25 |
| Tanh | (ez−e−z)/(ez+e−z) | (−1, 1) | 1 − tanh², max 1 |
| ReLU | max(0, z) | [0, ∞) | 0 (z<0) or 1 (z>0) |
| Softmax | ezᵢ/Σezⱼ | probabilities, sum 1 | (with CE loss: p − y) |
Why Nonlinearity Is Non-Negotiable: XOR
Compose two linear layers with no activation: W₂(W₁x + b₁) + b₂ = (W₂W₁)x + (W₂b₁+b₂) — still one linear map. Depth buys nothing without nonlinearity. The classic witness is XOR: points (0,0), (1,1) in one class and (0,1), (1,0) in the other cannot be split by any single line (Minsky & Papert’s perceptron limitation), but a 2-unit hidden layer with a nonlinear activation carves the plane into the two diagonal regions easily. “Minimum hidden units to solve XOR?” — 2 — is a recurring one-marker, the same separability story told for kernels in the SVM guide.
A Forward Pass, Computed by Hand
Network 2–2–1, all sigmoid. Input x = (1, 0). Hidden unit 1: weights (0.5, 0.5), bias 0. Hidden unit 2: weights (1, −1), bias 0. Output unit: weights (1, 1), bias −1.
Hidden layer: z₁ = 0.5(1) + 0.5(0) = 0.5 → h₁ = σ(0.5) = 0.622. z₂ = 1(1) − 1(0) = 1 → h₂ = σ(1) = 0.731.
Output layer: zout = 1(0.622) + 1(0.731) − 1 = 0.353 → ŷ = σ(0.353) ≈ 0.587.
Every GATE forward-pass NAT is this pattern: weighted sum, add bias, apply the named activation, repeat layer by layer. Keep σ(0.5) ≈ 0.622, σ(1) ≈ 0.731, σ(2) ≈ 0.881 handy — the same sigmoid anchors from the logistic regression guide.
Backpropagation and One Step of Gradient Descent
Backpropagation is the chain rule, organised. The loss depends on the output, the output on the last layer’s weights and inputs, those inputs on the previous layer, and so on — so ∂L/∂w for any weight is a product of local derivatives along the path from that weight to the loss. Computing these products backward layer by layer reuses shared factors, which is why one backward pass costs about as much as one forward pass.
Vanishing gradients: with sigmoid activations every chain factor includes σ′ ≤ 0.25, so a 10-layer chain shrinks gradients by up to 0.2510 ≈ 10−6 — early layers barely learn. This is the standard MCQ answer for “why ReLU in deep networks?”: its derivative is 1 for positive inputs, so gradients survive depth.
Gradient descent, one worked step: take L(w) = (w − 3)², w₀ = 0, learning rate η = 0.25. Gradient: ∇L = 2(w−3) = −6 at w₀. Update: w₁ = 0 − 0.25(−6) = 1.5. Next step: ∇L = 2(1.5−3) = −3, so w₂ = 1.5 + 0.75 = 2.25 — halving the gap to the optimum each step. Too large an η overshoots and diverges; too small crawls — the tuning trade-off connects directly to the bias–variance and validation machinery.
Softmax + Cross-Entropy, Computed
Output logits z = (2, 1, 0). Exponentials: e² = 7.389, e¹ = 2.718, e⁰ = 1. Sum = 11.107. Softmax probabilities: p = (7.389, 2.718, 1)/11.107 = (0.665, 0.245, 0.090) — they sum to 1, and the largest logit gets the largest probability (softmax preserves ranking; subtracting a constant from every logit changes nothing).
If the true class is the first, cross-entropy loss = −ln(0.665) ≈ 0.408. And the elegant fact that makes deep learning trainable: with softmax + cross-entropy, the output-layer gradient is simply p − y = (0.665−1, 0.245, 0.090) = (−0.335, 0.245, 0.090).
Three GATE-Style Problems, Solved
Problem 1 (NAT). How many trainable parameters (with biases) in a 10–20–15–5 MLP?
Solution. 10×20+20 = 220; 20×15+15 = 315; 15×5+5 = 80. Total = 615.
Problem 2 (NAT). Network 2–2–1 with ReLU hidden units, linear output. x = (2, −1). Hidden: h₁ weights (1, 2), bias −1; h₂ weights (0, 1), bias 2. Output weights (3, 2), bias 0. Find the output.
Solution. z₁ = 2 − 2 − 1 = −1 → ReLU → 0. z₂ = −1 + 2 = 1 → 1. Output = 3(0) + 2(1) + 0 = 2. Note how ReLU killing h₁ simplified everything — watch for that design in exam questions.
Problem 3 (MCQ). Which is TRUE? (a) sigmoid outputs can be negative; (b) tanh outputs lie in (−1, 1); (c) ReLU’s derivative at z = −2 is 1; (d) softmax outputs can exceed 1.
Solution. (b). Sigmoid is strictly positive; ReLU’s derivative is 0 for negative inputs (the “dying ReLU” region); softmax entries are probabilities in (0, 1).
Common Mistakes to Avoid
Forgetting biases — or including them when the question says “weights only”. The two answers differ by exactly the total number of non-input units.
Counting the input layer as parameterised. It isn’t; parameters live on connections into a layer.
Applying the activation before adding the bias. Order is: weighted sum → + bias → activation.
Sign slips in the update rule. w ← w minus η∇L — you descend the gradient, not climb it.
Normalising softmax with a sum of logits instead of exponentials. Exponentiate first, then divide by the sum of exponentials.
How GATE DA Asks Neural Networks
Recurring shapes: (1) NAT — parameter/connection counting for a stated architecture (the most predictable 1–2 marks in the paper); (2) NAT — a small forward pass, often with ReLU zeroing one unit, or one gradient-descent step; (3) NAT — softmax probabilities or a cross-entropy value; (4) MCQ — activation ranges/derivatives, vanishing gradients, why nonlinearity/XOR. Arithmetic stays small; layout discipline wins the marks.
Master the full ML syllabus for GATE DA 2027
My complete Machine Learning course covers neural networks, backpropagation, regression, SVM, trees, clustering and PCA with recorded lectures, notes and GATE-level practice — aligned exactly to the DA syllabus.
Explore the Machine Learning Course →FAQs: Neural Networks for GATE DA
Do I need CNNs, RNNs or transformers for GATE DA?
The core syllabus centres on feedforward networks (MLPs), activations and training. Skim what convolution does conceptually, but invest your depth in parameter counting, forward passes and backprop-level understanding.
Do I have to derive backpropagation fully?
No — GATE tests the chain-rule idea, the p − y softmax gradient, one-step gradient-descent updates and vanishing-gradient reasoning, not multi-page derivations.
Why does everyone use ReLU instead of sigmoid in hidden layers?
Sigmoid’s derivative caps at 0.25, so deep chains of sigmoids shrink gradients exponentially. ReLU passes gradient 1 for positive inputs, is cheap to compute, and empirically trains much faster — at the cost of possible “dead” units.
How do neural networks relate to logistic regression?
A single sigmoid output unit with no hidden layer IS logistic regression. Hidden layers learn the features; the output layer is the familiar linear-plus-sigmoid from the regression comparison guide.
What should I study next?
You’ve completed the ML deep dives — move to the programming section (sorting and graph traversals in Python) or consolidate with the ML pillar guide‘s revision checklist.
Round out your prep: revisit regularisation and evaluation in the bias–variance & cross-validation guide, connect the sigmoid story back to logistic regression, and track your coverage against the GATE DA 2027 syllabus. New ML problem-solving sessions drop regularly on my YouTube channel — subscribe so you don’t miss them.
Recent Post
A* search and alpha-beta pruning for GATE DA 2027: full open/closed-list trace, admissible heuristics, minimax with pruning counted leaf by leaf, solved problems.
Taylor series and maxima-minima for GATE DA 2027: standard expansions, e^0.1 and cos(0.2) approximated, derivative tests and the Hessian rule worked with solved problems.
Normal forms for GATE DA 2027: functional dependencies, attribute closure worked, 1NF to BCNF with full decompositions, checklist table and solved GATE problems.
SQL and relational algebra for GATE DA 2027: σ, π and joins worked on sample tables, GROUP BY and nested queries evaluated row by row, plus solved GATE problems.
BFS and DFS for GATE DA 2027: graph traversals traced step by step with queue and stack states, shortest paths, Python code, complexity and solved GATE problems.
Sorting algorithms in Python for GATE DA 2027: bubble, insertion, selection, merge and quick sort traced step by step, binary search, complexity table and solved problems.
Learn Daily, Wherever You Are
Free lectures, exam updates, PYQ discussions, and job alerts — delivered through our YouTube channel and Telegram communities.
