Building a Tiny GPT - Part 3: Making a Number Learn
- Published on
In the previous post, I built a simple text generator by counting which characters appeared next to each other. It could produce some words, occasionally... But there was still a fair way to go before it could produce anything useful.
Before going any further with text, I want to understand what actually happens when we train a neural network. How does getting an answer wrong help it get the next one right?
So for this part, I am going back to some fairly simple numbers and a problem small enough that I can check the answers myself.
Part 3: Making a Number Learn
I will be starting with a single neuron. For now, this is just a calculation with two numbers I can adjust: a weight and a bias. I'll get into what those do shortly.
The idea is to give it a few examples and let it adjust those numbers until its answers get closer to what I expect. If I can follow that, I should have something to build on when there are more neurons involved.
There are a few maths terms coming up here, and part of the point of this exercise is to understand what they mean in the code. I want to be able to follow why the numbers change, rather than just have something that appears to work.
Starting With Something Simple
I am using five training examples:
Input x |
Target y |
|---|---|
| -2 | -3 |
| -1 | -1 |
| 0 | 1 |
| 1 | 3 |
| 2 | 5 |
The rule behind these is y = 2x + 1. In other words, take the input, multiply it by two, and add one. Nothing too exciting so far...
I know the rule because I used it to create the examples. But the training code only gets the input and expected answer for each one. It has to work towards the right weight and bias from there, starting with both at zero.
The text files and tokenizer can sit this one out. There is enough to get my head around here without those as well.
The Neuron
The calculation looks like this:
prediction = weight * input + bias
The weight is the number I multiply the input by, and the bias is the number I add afterwards. These are the model's parameters, which is just the name for the values training will adjust.
So if training works, I should end up with a weight close to 2 and a bias close to 1. Giving it 3 would then produce 2 * 3 + 1 = 7.
For now, with both values at zero, it will predict zero for every input. Not a particularly good start, but at least there is room for improvement!
This is called a linear neuron because the answers it can produce form a straight line when plotted against the inputs. There isn't another transformation, or activation, applied to the result here. I'll come back to those when I need more than a straight line.
In LinearNeuron.Predict, the calculation is simply:
var prediction = Weight * input + Bias;
Making a prediction does not change the weight or bias. I need a separate training step for that, but first I need a way to tell how wrong the prediction was.
Measuring How Wrong It Is
I will use what is called mean squared error. The name is a bit of a mouthful, but the calculation is fairly straightforward: work out the difference between each prediction and its expected answer, multiply that difference by itself, then take the average across the examples.
error = prediction - target
loss = sum(error * error) / numberOfExamples
For example, predicting 0 when I wanted 5 gives an error of -5. Squaring it gives 25. An error of 5 also gives 25, so being too high and being too low cannot cancel each other out when I add them together. Bigger mistakes also add more to the score.
With the weight and bias still at zero, I get:
| Input | Target | Prediction | Error | Squared Error |
|---|---|---|---|---|
| -2 | -3 | 0 | 3 | 9 |
| -1 | -1 | 0 | 1 | 1 |
| 0 | 1 | 0 | -1 | 1 |
| 1 | 3 | 0 | -3 | 9 |
| 2 | 5 | 0 | -5 | 25 |
The last column adds up to 45. Divide by the five examples and I get 9. This is the loss, the score I want training to bring down.
This is a different calculation to the loss in part 2, so comparing the two scores wouldn't tell me anything useful. Here, I just want to see whether this score goes down as the predictions improve.
Which Way Should the Numbers Move?
Knowing how wrong the answers are is useful, but I still need to know what to change. Should the weight go up or down? And by how much?
This is where gradients come in. For each parameter, I calculate how sensitive the loss is to a small change in that value. A positive result means increasing it a little would increase the loss; a negative result means increasing it a little would decrease the loss.
The maths term for each of those calculations is a derivative. For this neuron, the formulas for one example are:
weightGradient = 2 * error * input
biasGradient = 2 * error
The 2 * error comes from squaring the error when calculating the loss. The weight calculation also includes the input because the weight is multiplied by it. The bias is just added, so it doesn't need that extra multiplication.
More formally, the derivative of error² is 2 * error. I then multiply by how the error changes with the parameter: input for the weight, or 1 for the bias. Joining those pieces together is called the chain rule. Keeping the calculation this small means I can see those pieces directly in the code.
For the example x = 2, target = 5, with zero parameters:
prediction = 0
error = 0 - 5 = -5
weightGradient = 2 * (-5) * 2 = -20
biasGradient = 2 * (-5) = -10
Both results are negative, so increasing the weight and bias a little should improve this particular answer. That is useful, but I have four other examples to consider too.
To take all five into account, I add up their results and divide by the number of examples, just as I did for the loss. In Evaluate, this looks like:
foreach (var (input, target) in examples)
{
var error = Predict(input) - target;
loss += error * error;
weightGradient += 2 * error * input;
biasGradient += 2 * error;
}
loss /= examples.Length;
weightGradient /= examples.Length;
biasGradient /= examples.Length;
With the weight and bias at zero, this gives me -8 for the weight gradient and -2 for the bias gradient. Evaluate returns those alongside the loss, ready for the training step.
Taking the First Step
Now I have something I can use to adjust the numbers. The update is:
newParameter = oldParameter - learningRate * gradient
Subtracting a negative gradient increases the parameter, and subtracting a positive one decreases it. That moves me in the direction that should reduce the loss, provided I don't take too big a step.
The learning rate controls the size of that step. This process is called gradient descent, so that is what the term means in this example: calculate which way to move, take a step, and repeat.
Using a learning rate of 0.1, the first update is:
weight = 0 - 0.1 * (-8) = 0.8
bias = 0 - 0.1 * (-2) = 0.2
In TrainStep, that becomes:
var result = Evaluate(examples);
var weight = Weight - learningRate * result.WeightGradient;
var bias = Bias - learningRate * result.BiasGradient;
Both gradients come from the same call to Evaluate, before either parameter changes. Once I have checked that the new values have not overflowed, I assign them to Weight and Bias.
After that first update, the loss drops from 9 to 3.52. Still wrong, but considerably less wrong than before! Now I can repeat the process with the new weight and bias.
I use all five examples for every update, which is what makes this full-batch gradient descent. Starting both parameters at zero is fine for this example, although I shouldn't assume that will work when I get to networks with hidden layers.
Checking My Maths
Of course, getting a lower loss doesn't prove I've got those formulas right. I want a separate way to check them before building anything bigger on top.
One way to do this is to change the weight by a tiny amount, see what happens to the loss, and compare that with what the gradient said should happen. I try a little above and a little below the current value, keeping the bias the same:
numericalWeightGradient = (loss(weight + epsilon, bias)
- loss(weight - epsilon, bias)) / (2 * epsilon)
epsilon is just the size of that tiny change. Here I use 0.000001, also written as 1e-6.
This estimate is called a central finite difference. I do the same for the bias, this time keeping the weight fixed. Temporary neuron instances let me try those changes without modifying the neuron I am training.
I compare these estimates with the gradients calculated by the formulas. For this example, I allow a difference of up to 0.000001. They are estimates using floating-point numbers, so I don't expect exact equality, and making the change smaller and smaller won't necessarily make the check more accurate.
The training loop still uses the formulas. This extra calculation is just there to check my working.
Watching It Learn
From the model repository root, I can run:
dotnet run -- neuron
The default demonstration runs 100 updates with a learning rate of 0.1:
One linear neuron (prediction = weight * input + bias).
Training pairs: (-2, -3), (-1, -1), (0, 1), (1, 3), (2, 5)
Learning rate: 0.1; updates: 100; initial weight: 0; initial bias: 0
Initial analytic gradients: weight=-8.000000, bias=-2.000000
Initial numerical gradients: weight=-8.000000, bias=-2.000000
Gradient check: PASS (epsilon=1E-6, absolute tolerance=1E-6)
Step | Weight | Bias | Mean squared error
0 | 0.000000 | 0.000000 | 9.000000E+000
1 | 0.800000 | 0.200000 | 3.520000E+000
10 | 1.987907 | 0.892626 | 1.182171E-002
25 | 1.999994 | 0.996222 | 1.427254E-005
50 | 2.000000 | 0.999986 | 2.037036E-010
100 | 2.000000 | 1.000000 | 4.149520E-020
Unseen input: x=3; prediction=7.000000; reference=7
Inference uses the learned parameters without changing them. This demo does not process text.
And there we have it. The weight gets closer to 2, and the bias gets closer to 1, without me assigning either of those values in the training loop.
By the last row, they round to the expected values at six decimal places. The loss is printed in scientific notation because it has become too small to see otherwise. For example, 2.037036E-010 is about 0.000000000204. Very small, but still not quite zero.
The final prediction uses 3, which wasn't in the training examples, and gives me 7 to the precision shown. This is inference: using the values learned during training to produce an answer, without changing them again.
Of course, I deliberately gave it examples from a straight line and a calculation that can represent one. It doesn't mean it can learn any rule I throw at it. But it does give me a training loop where I can follow what is happening from start to finish.
What Happens If I Change It?
So what happens if I change how big a step it takes?
dotnet run -- neuron --rate 0.01
dotnet run -- neuron --rate 0.1
dotnet run -- neuron --rate 0.6
With the same starting values and 100 updates, 0.01 makes slower progress, while 0.1 gets close to the answer. At 0.6, the numbers grow wildly instead of settling down. The steps are too big, and the predictions get worse.
So simply turning up the learning rate won't get me to the answer faster...
Another thing to try is temporarily removing the bias from the prediction and its update. Would the weight alone be enough?
The example (0, 1) makes the problem fairly clear. Whatever the weight is, multiplying it by zero will still give zero. I need the bias to get an answer of 1 for that input. Without it, there isn't a weight that can fit all five examples, however long I train for.
Making Sure It Works
As with the previous parts, I have checks for more than just the final answer. These include the starting loss of 9, the gradients of -8 and -2, and the first update to 0.8 and 0.2. I also compare the gradients with the numerical estimates at a few different weights and biases.
There are checks that predictions don't change the parameters, that both updates use gradients from before either value changes, and that training gets close to the expected answers. Empty data, invalid learning rates, and numbers that overflow are covered too. The full implementation includes those checks around the calculations shown above.
Summary
In this post, I trained one neuron to work out a simple rule from five examples. I started with a weight and bias of zero, measured how wrong the predictions were, and used that to work out which way to adjust the two values.
The useful part for me is being able to follow how an incorrect prediction leads to a change in the code's numbers. There are only two of them here, but this gives me somewhere to start when there are more involved.
Next, I will add a hidden layer and a nonlinear activation to try learning XOR, a rule that a single straight line can't represent. A little more to get my head around, but at least I now have something working to build on.