How Neural Networks Learn: Loss, Gradients, Backprop and Gradient Descent

Training a neural network sounds mysterious, but it is really one short loop repeated many times. You can say the whole thing in a single breath:

Make a prediction, see how wrong it is, turn that wrongness into one number (the loss), use calculus to find how each weight affected that number (the gradient, found by backpropagation), nudge every weight in the direction that lowers the loss (gradient descent), and run it again. The predictions get better.

That is the entire idea. Let us walk each link of the chain, then watch a tiny network actually learn.

1. Prediction

A network takes an input and produces an output. At the start, its internal numbers (the weights) are random, so the output is basically a guess. If we ask it for 2 × 5 + 1 it might say 0.3. That guess is the prediction.

2. Error

We compare the prediction to the correct answer. The difference is the error. If the right answer was 11 and the model said 0.3, the error is large. One prediction gives one error.

3. Loss

We do this for many examples and roll all those errors into a single number: the loss. A common choice is mean squared error (square each error so negatives do not cancel, then average). The loss answers one question: overall, how wrong is the model right now? Lower is better, and the whole point of training is to make this number small.

4. Gradient

Here is where calculus enters, gently. For each weight, we ask: if I nudge this weight up a little, does the loss go up or down, and how fast? That answer, for every weight at once, is the gradient. It is a direction: it points the way the loss increases fastest. So the opposite direction is the way to make the loss go down.

5. Backpropagation

A deep network has millions of weights stacked in layers, so computing the gradient for each one naively would be impossibly slow. Backpropagation is the clever shortcut: start at the loss at the output, and use the chain rule from calculus to pass the error backward through the layers, computing each weight's gradient along the way. It reuses work from later layers to get earlier ones, so it finds every gradient in roughly one backward pass.

This is the precise version of a phrase people often blur: backprop does not change the weights. It only computes the gradients. The changing happens next.

6. Gradient descent

Now we have, for every weight, the direction that would lower the loss. Gradient descent takes the step: move each weight a little bit in the opposite direction of its gradient. How big a step is set by the learning rate. Too small and learning crawls; too big and it overshoots and never settles.

7. Repeat

One pass does not fix much. But run the loop thousands of times and the loss keeps dropping, step by step, until the predictions are good. That is training.

Watch it learn

Here is the whole loop in plain Python, with no libraries. The model is as simple as it gets, prediction = w*x + b, and we ask it to learn the rule behind the data, which is y = 2x + 1. It starts knowing nothing (w = 0, b = 0).

xs = [0, 1, 2, 3, 4]
ys = [1, 3, 5, 7, 9]            # the true answers (2x + 1)
 
w, b = 0.0, 0.0                 # start knowing nothing
lr = 0.05                       # learning rate
n = len(xs)
 
def mse(w, b):
    return sum((w*x + b - y)**2 for x, y in zip(xs, ys)) / n
 
for step in range(401):
    # gradient of the loss for w and b
    gw = sum(2*(w*x + b - y)*x for x, y in zip(xs, ys)) / n
    gb = sum(2*(w*x + b - y)   for x, y in zip(xs, ys)) / n
    # gradient descent: step each weight against its gradient
    w -= lr * gw
    b -= lr * gb
 
print(w, b)

Printing the loss and weights along the way, this is the real output:

step       loss       w       b
   0    33.0000   0.000   0.000
   1     3.6100   1.400   0.500
   5     0.0180   2.072   0.769
  20     0.0070   2.051   0.855
 100     0.0001   2.005   0.987
 400     0.0000   2.000   1.000
 
learned:  w = 2.000, b = 1.000   (true answer: w = 2, b = 1)
predict x=10 ->  21.00   (correct: 21)

Read it top to bottom and you can see the chain working. The loss starts at 33 (the model is badly wrong), and every step the gradient points the way down, gradient descent takes a step, and the loss falls, fast at first, then fine-tuning. By the end the model has discovered w = 2 and b = 1 on its own, and it correctly predicts an input it was never given: x = 10 gives 21.

Summary

• A network learns by repeating one loop: predict, measure error, roll it into a loss, find the gradient, step the weights, repeat.
• Loss is one number for how wrong the model is; training makes it small.
• Backpropagation computes the gradient for every weight using the chain rule.
• Gradient descent uses those gradients to nudge the weights; the learning rate sets the step size.
• The tiny example above learned y = 2x + 1 from scratch, and the same loop trains the largest models.

Next, see where this loop runs at massive scale in What is an LLM? — the training section there is this same idea with billions of weights.