How does backpropagation compute every gradient?
What interviewers are testing
This question tests whether you understand training as a computational graph rather than a library call. Interviewers listen for the chain rule stated precisely, for why activations are cached, and for the intuition that depth turns gradients into long products that can vanish or explode. The follow-ups — reverse versus forward mode, residual connections, gradient accumulation — show whether the model extends to the debugging questions that real training runs generate.
Mental model
The forward pass computes activations and a scalar loss, caching everything the derivatives will need. The backward pass walks the graph in reverse, letting each layer multiply the gradient it receives by its own local Jacobian, so every weight ends up with dL/dW. An optimizer then nudges each parameter along the negative gradient scaled by the learning rate. Depth makes that backward product long, which is why gradient magnitudes need residual paths, normalization, and careful initialization to stay near one.
Step-by-step solution
Step 1 of 5
Forward pass computes the loss
Training starts with a forward pass. The input batch flows through each layer: a linear transform followed by a nonlinearity, producing activations a1, a2, and finally the output prediction. The loss function compares that prediction with the target and reduces the difference to a single scalar — cross-entropy for classification, mean squared error for regression. Watch the animation as the signal moves left to right, one node at a time, until the scalar loss appears at the end. The crucial detail is what the framework does silently: every intermediate activation and pre-activation is cached, because the backward pass will need those exact values as multiplicative factors. Without the cache, gradient computation would have to recompute the forward pass; with it, each local derivative can be evaluated cheaply. That cached graph of tensors is the computation graph autograd builds, and the scalar loss is its root.
Animation — Forward pass computes the loss
a1 = σ(z1)
a2 = σ(z2)
scalar
The input batch enters; weights and biases are the parameters to be learned.
Edge cases & traps
- Forgetting to zero gradients between steps: PyTorch accumulates into .grad, so two iterations without zero_grad() sum two batches of gradients — call optimizer.zero_grad() every step.
- Blowing up memory by keeping the whole autograd graph alive: detach tensors used only for metrics and wrap evaluation in torch.no_grad(), or the graph pins activations for the entire run.
- Debugging a silent vanishing signal by watching the loss alone: log per-layer gradient norms (hooks or a profiler) so a decaying backward signal is visible before training plateaus.
- Clipping gradients and declaring victory: clipping caps NaNs but does not fix a bad init or saturated activations — pair it with residual and normalization fixes.
- Choosing a learning rate too large for the loss surface: the step overshoots and the loss diverges even though gradients are correct — sweep the LR or add warmup and a schedule.