Building a Tiny GPT - Part 4: A Network to Learn XOR

Published on

So in the last part, I built a single linear neuron that learned a straight-line rule. I could trace a prediction through its error, derivatives, and parameter updates. But what do we do when the target cannot be represented by a linear model?

For this post, I'm going to build a tiny neural network to learn the XOR function.

Part 4: A Network to Learn XOR

An XOR function outputs 1 when two input bits differ, otherwise it outputs 0. The truth table is:

Input 1 Input 2 Output
0 0 0
0 1 1
1 0 1
1 1 0

The real question is whether our new model can fit this complete table, rather than whether it can generalize from a subset. In other words, can it learn the XOR function exactly from the XOR truth table, and only the XOR function?

Let the Linear Model Try First

Before we go ahead and add the extra complexity of a hidden layer, let's see how the linear model from the last part performs...

Because this time it has two inputs, the prediction is w1 * x1 + w2 * x2 + bias. We can train it on the four rows above using the same mean squared error as the previous post.

We can already see where this goes wrong by wroking through the table. When both inputs are 0, the prediction is just the bias, so the bias has to be 0. When only one input is 1, the prediction is just taht input's weight. XOR gives 1 in both of those cases, so both weights have to be 1. But if you put 1 into both inputs, we get 2 when the table says we need 0.

If I run the training longer, the model settles near 0.5 for all four inputs, with a mean squared error of 0.25. Training for longer can't fix the problem so we have to change our approach. The XOR example in Deep Feedforward Networks provides background on this capacity limitation and the role of hidden transformations.

Introducing Tanh

Introducing tanh, short for hyperbolic tangent (and pronounced "tanch"), is a nonlinear activation function that squashes its input to the range [-1, 1]. It is defined mathematically as:

tanh(x) = (e^x - e^{-x}) / (e^x + e^{-x})

The useful part is the shape of that curve. Near zero, a small change in the input makes a noticeable difference. Further out, the curve flattens. That bend is what makes tanh nonlinear. It will let our hidden neurons transform the inputs in different ways, so the output neuron can combine their results to learn XOR.

Adding Four Non-Linear Hidden Units

Each hidden neuron adds up its weighted inputs and bias, then passes the result through tanh. An input of 0 gives 0. As the input gets more positive, the output approaches 1, and as it gets more negative, the output approaches -1.

h_j = tanh(w1_j * x1 + w2_j * x2 + bias_j)
prediction = outputBias + sum(outputWeight_j * h_j)

The nonlinearity matters. Composing affine layers without it would still produce an affine function of the inputs. Hence, the hidden units can learn a transformation that makes the final task easier for the output layer.

I initialize input and out weights with seeded random values and leave biases at zero. Starting every weight at zero would keep the hidden units inactive in this implementation. Only the output bias could learn, leaving the network predicting the same value for every row.

Following Errors Backwards

The output derivative begins as in the last part: 2 * (prediction - target) / batchSize. An output weight receives that derivative multiplied by its hidden activation. The hidden unit receives it multiplied by its output weight, then by the derivative of tanh:

gradients[12 + unit] += outputGradient * hidden[unit];
var hiddenGradient = outputGradient * _parameters[12 + unit] * (1 - hidden[unit] * hidden[unit]);
var offset = 3 * unit;
gradients[offset] += hiddenGradient * x1;
gradients[offset + 1] += hiddenGradient * x2;
gradients[offset + 2] += hiddenGradient;

For a small example calculation, take hidden activation 0.5, output weight 2, prediction error -1, and batch size 4. The output derivative is -0.5, the output-weight contribution is -0.25, and the derivative reaching the hidden preactivation is -0.75. At input (1,0), the two input-weight contributions are -0.75 and 0.

This is the chain rule applied along the computation's dependencies. The hidden units have no target labels of their own. Their updates depend on their contribution to the output error. I accumulate the full batch's gradients before changing any parameter.

See what the network learns

dotnet run -- xor

Captured output:

Phase 5: XOR with a 2-4-1 network (tanh hidden units, linear output).
Seed: 42 | learning rate: 0.1 | full-batch updates: 10000
Trained linear baseline MSE: 0.250000
Gradient check (17 parameters, epsilon=1E-6, absolute tolerance=1E-6)
Step | Network mean squared error
0 | 8.009137E-001
1 | 6.118386E-001
100 | 2.211973E-001
1000 | 1.105046E-012
10000 | 4.938277E-030
x1 x2 | Target | Linear | Network | Network class
0 0 | 0 | 0.500000 | 0.000000 | 0
0 1 | 1 | 0.500000 | 1.000000 | 1
1 0 | 1 | 0.500000 | 1.000000 | 1
1 1 | 0 | 0.500000 | 0.000000 | 0
Network accuracy: 4/4 (threshold 0.5)
Network outputs are scores, not probabilities. No text datasets are used.

The network starts with a higher loss than the fitted linear baseline, then learns all four cases. The scores round to zero and one in the table, with a small residual loss visible above. I classify scores at 0.5 - the linear output is not constrained to return a probability.

This is a successful fit on four known inputs. It is a useful check that the architecture and learning calculation work together, while saying little about behaviour away from those points. Different seeds may also change how training progresses.

Making Sure It Works

Given we have a neural network now, to demo this, we can finally start to visualize the network's learning.

Choose an input pair and move the slider to explore training.

XOR network activations Two inputs connect to four tanh hidden units, which connect to one linear output. Solid connections have positive weights; dashed connections have negative weights. Thicker connections have larger absolute weights.
Solid: positive weight · dashed: negative weight · thickness: weight magnitude
InputsTargetScoreClass ≥ 0.5
Seed 42 · learning rate 0.1 · all four cases are training data

The Learning Curve

1.0001,000 steps

The solid line shows how the network's mean squared error changes over the first 1,000 training steps. The dashed line marks 0.25, where the linear model settled. Our network actually starts out worse, with an error of about 0.80, but it drops below that line early in training.

The curve flattens for a bit, then drops again. By 1,000 steps the error is close enough to zero that you can barely see it on this scale. If you move the slider above to that checkpoint, you can see what that looks like for each input pair. The scores are close to 0 or 1, and all four land on the correct side of the 0.5 threshold.

Summary

In this post, I built a small neural network to learn the XOR function. I demonstrated how hidden layers and nonlinear activations enable the network to learn complex patterns that a single linear neuron cannot. I also provided a visualization of the network's learning process to show how the network's predictions evolve over training updates.

In the next part, I will move onto the next step of building a BPE tokenizer, which will allow us to handle larger vocabularies and more complex text inputs.

Stay tuned!