BPE tokenizer from scratch

This is the start of the deep-implementation phase. Rules from build it, don't copy it. So I built a BPE tokenizer from scratch, the thing that turns text into numbers before any model ever sees it.

And honestly? It's the most underrated piece of the whole stack. Everyone obsesses over attention and quantization, but the tokenizer is where your model's entire vocabulary of thought is defined.

Why BPE

Words are too coarse (you'd never cover all of English, let alone code or Hindi) and characters are too fine (a model can't learn "th" is a useful unit from single letters). Byte-pair encoding splits the difference: start with bytes, and repeatedly merge the most frequent adjacent pair. The result is a vocabulary of subword units that are exactly as fine as the data demands.

The algorithm, in one paragraph:

1. Start with a corpus of text, split into bytes (or Unicode codepoints).
2. Count every adjacent pair.
3. Merge the most frequent pair into a new token.
4. Repeat until you reach your target vocabulary size.

That's it. That's the whole trick. The vocabulary is just a list of "these two things usually appear together, so let's make them one thing".

Building it

Here's the core of my implementation, the merge loop:

def learn_bpe(corpus, vocab_size):
    # start from bytes
    tokens = [list(bytearray(s.encode('utf-8'))) for s in corpus]
    merges = []
    while len(set(b for t in tokens for b in t)) + len(merges) < vocab_size:
        # count adjacent pairs
        pairs = Counter()
        for t in tokens:
            for a, b in zip(t, t[1:]):
                pairs[(a, b)] += 1
        if not pairs: break
        (a, b), _ = pairs.most_common(1)[0]
        merges.append((a, b))
        # apply merge: replace every (a, b) with a new id
        new_id = 256 + len(merges) - 1
        for i, t in enumerate(tokens):
            nt = []
            j = 0
            while j < len(t):
                if j < len(t)-1 and t[j] == a and t[j+1] == b:
                    nt.append(new_id); j += 2
                else:
                    nt.append(t[j]); j += 1
            tokens[i] = nt
    return merges

The subtle part is the new_id: bytes are 0-255, so every merged token gets an id above 255. The ids stay stable across the whole training, which is what makes the vocabulary shareable.

What I learned building it

• Merges are greedy and local. BPE never looks ahead. It merges the most frequent pair in the current corpus, and that's it. This is both its strength (simple, deterministic) and its weakness (no global optimization).
• Frequency is everything. The first few merges are almost always whitespace-related, because " t", " th", " the" are the most common byte pairs in English text.
• It's language-agnostic. Feed it Hindi or code and it finds the right units on its own. That's why BPE won.

Why this matters for inference

Here's the inference angle: the tokenizer determines how many tokens your prompt becomes, and tokens are what cost money and time. English is roughly 1 token per 4 characters. But the same sentence in a language with a worse tokenizer can be 2-3x more tokens, which means 2-3x more KV cache, more decode steps, more cost.

When someone says "our model is slow on Hindi", the first thing I check isn't the model. It's whether the tokenizer is wasting tokens. It happens constantly.

The takeaway

You can't optimize what you can't tokenize. The tokenizer is the first bottleneck, and it's the one everyone forgets.

Tomorrow: the autoregressive decoder loop in PyTorch, the actual engine of generation.