Building a Tiny GPT - Part 2: Generating Text From Numbers

Published on

Having established a reversible numerical representation of text, I will now use those character IDs to make predictions about what might follow a given character. This will allow me to produce a simple text generator.

Part 2: Turning Numbers into Text

For this pass, I will use a count table. It should be small enough calculate by hand, but it gives me three useful pieces of a language-model experiment: probabilities, generated text, and a prediction score.

In part 1, I added a text file with facts the model will train on. I have also added a separate validation text file to answer the question, "Did the model learn character-pair patterns that also help on another piece of text?":

Mira carries apples to the silver bridge.
Tavi carries pears to the stone tower.

This is a small, closely related sample. It demonstrates evaluating text that did not supply the counts. It does not establish general language ability or leave me with an untouched final test set if I repeatedly tune against it.

The Count Table is the Model

For a worked example, given the vocabulary is [a, b, c] and the training text is ababa. Its four adjacent pairs are ab, ba, ab, and ba. That gives two occurrences in each direction and zero for every other pair.

CountedBigram counts adjacent token IDs directly:

for (var i = 1; i < trainingTokens.Length; i++)
{
    var previous = trainingTokens[i - 1];
    var next = trainingTokens[i];
    _counts[previous, next]++;
    _totals[previous]++;
}

The final a has no next character, so its presence does not add an outgoing transition. I count only actual adjacent pairs - I do not wrap the end of the file back to its beginning.

Give Unseen Pairs a Probability

I use add-one smoothing to turn each row into a probability distribution:

public double Probability(int previous, int next) =>
    (Count(previous, next) + 1.0) / (_totals[previous] + (double)VocabularySize);

In the example, the row after a is [1/5, 3/5, 1/5]. The probabilities sum to one, and even an unseen pair such as aa has a nonzero probability. A context with no outgoing counts gives a uniform row.

This is the character-level version of the bigram approach described in Jurafsky and Martin's N-gram Language Models. I am using one previous token as the context, with a deliberately simple smoothing rule.

A Score for the Actual Next Character

For each pair being evaluated, I take the negative natural logarithm of the probability assigned to the actual next token, then average over the pairs. A probability of 3/5 gives a loss of about 0.510826; a probability of 1/5 gives about 1.609438.

This makes an unlikely actual continuation more costly. I report the mean in nats per pair, alongside uniform guessing at ln(vocabularySize). The first character supplies context rather than receiving a score, so an N-token file contributes N - 1 predictions.

Testing the Model

This is the captured output with the default prompt Mira and seed 42:

Training: stories.txt (292 tokens, 291 pairs)
Validation: validation.txt (81 tokens, 80 pairs)
Vocabulary: 29 UTF-16 code units (training only)
Uniform baseline loss: 3.367296 nats/pair
Training loss: 2.420321 nats/pair
Validation loss: 2.632872 nats/pair
Next tokens after "a" (top 5):
" ": count=12, probability=0.209677
"s": count=4, probability=0.080645
"v": count=4, probability=0.080645
"r": count=3, probability=0.064516
"t": count=3, probability=0.064516
Seed: 42; new tokens: 120
Prompt: "Mira"
Generated: "MirarMMi. kgLs aiabak ngcM.iri i sikM amp\neslcM\n mi Tobl.\nhe bmiM\nTa i c\npeii\nr\n\n.g stdao a bphbavio\nsvcmNLgas NPikwrknPiwn\n"

“Nats per pair” means that penalty averaged over all adjacent-character pairs.

Both observed losses are below uniform guessing, and validation loss is higher than training loss. Those numbers describe these files under this model. The generated sample still looks very rough: knowing one character gives little context, and add-one smoothing puts probability onto pairs that never appeared.

To generate, I draw a number between zero and one, walk through the row's cumulative probabilities, and select the token whose interval contains the draw. The selected token becomes the next context.

There is a particularly clear limit here. Mira and Luma end in the same character, so they produce identical new continuations with the same seed. The code displays the whole prompt but conditions each prediction on just its last character. It has no representation of which person or place the prompt names.

Edge Cases

Cases covered in unit tests include known pair counts, the worked probabilities and loss, normalized rows, unseen contexts, seeded sampling, and invalid inputs.

A fixed-seed experiment draws 10,000 next tokens from one context and checks that frequencies are close to the expected probabilities. Scoring and generation must not change the fitted counts.

Unknown characters still fail at tokenization: smoothing can rescue a missing pair of known characters, but it cannot invent an entry in the vocabulary.

Summary

I now have my first language-model baseline built entirely from counts. In the next part, I will make one neuron learn with gradient descent, introducing the weight updates that later neural models will need.

I'll leave you with it's first words:

rMMi. kgLs aiabak ngcM.iri i sikM amp
eslcM
 mi Tobl.
he bmiM
Ta i c
peii
r

.g stdao a bphbavio
svcmNLgas NPikwrknPiwn

Isn't that just enough to bring tears to your AIs, huh?

PS: A Little More Training

Just for fun, let's train the model on the script of the Bee Movie, and prompt it with bees (again with a seed of 42). The generated text is:

haAl.
That be'ran the Onop r whe Heer louie Ifl boll, ne hiof f!
TBe tane t 1's wonde f hind wer tt's That adedyothanye 

You can begin to see the importance of training on a large dataset. The model has learned some patterns, and while it is capable of being able to produce some real words with a larger training dataset, it is still far from producing coherent text.

This is to be expected at this point of course, as we are only predicting at the character level. In the future, we will be able to predict at the word level, which will allow us to produce better outputs.