The unit the machine reads

Something small throws most people the first time they use AI: a model that can draft a whole contract miscounts the letters in a simple word. It is not weakness. It is that the model never saw the letters at all.

What tokenization is

Picture a workshop that only takes goods by the bale. You hand over a roll of cloth, and nobody measures it in centimetres: they cut it into bales of a familiar size and count the bales. Everything they then say about that roll, the price and the time it will take, is counted in bales.

Tokenization is that cut. Before reading anything, a machine chops the text into small pieces called tokens, usually a short word or part of a word, and turns each piece into a number. From then on it works only with those numbers, and no longer sees a single letter. Asking it how many letters are in a word is like asking that workshop how many centimetres of cloth they have.

How large the pieces are depends on the cutter. For English, the rule of thumb OpenAI gives is that one token is roughly four characters, or about three quarters of a word, so a hundred tokens come to about seventy-five words.

The more striking part is that the same content takes a very different number of pieces from one language to another. Ahia and colleagues, in work published at the EMNLP conference in 2023, found that some languages need up to five times as many tokens as English to say the same thing. Because the bill is counted in tokens, the same question costs one speaker more than another.

How this applies to running a business

What I see here is a question of units. The machine is not wrong, it is simply measuring in a unit other than the one we assumed it used. Ask how many letters while it counts pieces, and the answer will miss.

I find this reflects a good deal of what happens inside a company. One team reports in orders, another in revenue. Each unit is right within its own scope, but once they are put together into one picture the numbers will not add up, and the meeting turns into an argument about counting rather than about the work.

The unit you choose does not only record the result. It shapes how people go about reaching it.

In the numbers you look at each week, is there one that two people on your team are reading in two different units?

To your desk

The machine cuts text into pieces and bills by the piece, so anyone who overlooks the unit will find the invoice stranger than expected.

Something to try this week, about twenty minutes: take one figure that matters in your company, then ask two people who use it, separately, how they define it, where they count from, and what they subtract. If the two answers differ, that figure is not yet one figure. It is two figures sharing a name.