Building a Tiny GPT - Part 1: Character Tokenization

Published on

I want to fully understand how large language models work. And what better way to learn than to build a small one myself?

Goal for the Series

The plan is to build a tiny trainable GPT, with a working result at every step. The How LLMs Work repository provides the inspiration for the sequence: pattern matching, neural networks, tokenization, embeddings, and a transformer.

All code is open-source and available on GitHub.

Part 1 Goals

For part 1, I will be establishing our first fundamental concept: a reversible numerical representation of text. I will also establish a small world to inspect and a repeatable source of randomness for later experiments.

A Small World to Inspect

I am using an invented world so I can inspect every fact the eventual model has been given. The starting dataset is six lines:

Mira lives in Luma. Mira carries a red bag.
Tavi lives in Neri. Tavi carries a blue bag.
Mira has a cat named Pip. Tavi has a dog named Sol.
Luma has a silver bridge. Neri has a stone tower.
Mira grows apples. Tavi grows pears.
The bell in Luma rings at dawn. The bell in Neri rings at dusk.

There are 12 facts here. Later, I will use them to construct questions and distinguish familiar wording from new paraphrases.

Handling Randomness

I want randomness to be repeatable, so I will use a random number generator with a seed. This will allow me to inspect the results of a particular run and compare it to others.

For later experiments, repeatable randomness will be important for debugging and understanding the behavior of the model and whether a change in results came from a code change ro from a different random sequence.

Thankfully, System.Random can be seeded to produce repeatable sequences of random numbers, so I will use this when I need randomness.

Part 1: Turning Text into Numbers

The concept here is to represent text in a way that a model can use. So I need to build a character tokenizer that gives each distinct code unit an integer ID, converts text into a sequence of those IDs, and reconstructs the original text.

A "Vocabulary" is a Lookup Table

Suppose my source text is cabca. There are three distinct characters: a, b, and c. I sort them and assign IDs starting at zero:

Character Token ID
a 0
b 1
c 2

Using that table, cab becomes [2, 0, 1]. Looking up each ID in the other direction gives me cab again.

The IDs do not describe meaning. They are labels in a particular vocabulary. The same vocabulary gives consistent IDs even if I reorder the source text; adding a new character can shift the mapping.

We construct that table like so:

var characters = vocabularyText.Distinct().Order().ToArray();
Vocabulary = Array.AsReadOnly(characters);
for (var id = 0; id < characters.Length; id++)
    _tokenIds.Add(characters[id], id);

The dictionary supports encoding, and the read-only list supports decoding.

Being Precise About Characters

I am using a C# char as the token unit. That is a UTF-16 code unit, which does not always correspond to one displayed character. Many emoji take two code units; combining marks introduce another distinction between code units and what I see on screen. Microsoft's character-encoding guide explains those layers.

For this tokenizer, an emoji can therefore take two tokens. That is a deliberate simplification, and the checks make it explicit. I can round-trip the original sequence exactly, but arbitrary generated token sequences could split a surrogate pair. The bundled stories are ASCII, and this phase only represents existing text. Later, I will build byte-based BPE.

Running the Tokenizer

The output of the tokenizer is a sequence of token IDs. The command fits the tokenizer to the bundled stories and prints the vocabulary:

Vocabulary: 29 UTF-16 code units (from facts.txt)
0: "\n" (U+000A)
1: " " (U+0020)
2: "." (U+002E)
3: "L" (U+004C)
4: "M" (U+004D)
5: "N" (U+004E)
6: "P" (U+0050)
7: "S" (U+0053)
8: "T" (U+0054)
9: "a" (U+0061)
10: "b" (U+0062)
11: "c" (U+0063)
12: "d" (U+0064)
13: "e" (U+0065)
14: "g" (U+0067)
15: "h" (U+0068)
16: "i" (U+0069)
17: "k" (U+006B)
18: "l" (U+006C)
19: "m" (U+006D)
20: "n" (U+006E)
21: "o" (U+006F)
22: "p" (U+0070)
23: "r" (U+0072)
24: "s" (U+0073)
25: "t" (U+0074)
26: "u" (U+0075)
27: "v" (U+0076)
28: "w" (U+0077)
Input: "Mira"
Token IDs: [4, 16, 23, 9]
Decoded: "Mira"

The newline gets ID 0, followed by the space and full stop. Printing escaped strings makes those otherwise easy-to-miss entries visible. I can trace M, i, r, and a through the table and verify the encoded sequence myself.

The implementation builds the vocabulary from the facts even when I supply my own text. It does not add new characters from that argument. For example, tokenizing a string with the character Z fails because Z is absent from the source vocabulary.

I chose an explicit error instead of replacing unknown input with a placeholder. A replacement would lose information and prevent the exact round trip I'm trying to demonstrate. When I introduce held-out model evaluation, I will fit the vocabulary on training data rather than enlarging it with evaluation examples.

Checking More Than the Happy Path

Encoding looks up each code unit in the dictionary. Decoding appends the vocabulary entry for each ID to a StringBuilder, checking the ID's bounds first. Empty text maps to an empty array and back.

The checks include the hand-worked cabca example, the full dataset, whitespace, mixed casing, punctuation, accents, a combining mark, and an emoji. Unknown code units and invalid IDs must fail clearly.

Summary

In this post, I created a reversible numerical representation of text. I also established a small world to inspect and a repeatable source of randomness for later experiments. In the next post, I will use the character IDs to make predictions about what might follow a given character to produce a simple text generator.