What is Tokenization and How Does It Work?

What is Tokenization and How Does It Work?

No computer—from the laptops we use in our daily lives to the powerful servers in data centers—can actually understand words, sentences, or letters directly. For computers to understand human language, text must first be converted into small fragments, then into numbers, and ultimately into matrices.

Tokenization fundamentally refers to the process of splitting human language text into small fragments and subsequently converting them into numerical representations so that an LLM (Large Language Model) can understand them.

Although the word "tokenization" is mostly used to mean simply breaking text into words, tokenization in the LLM world encompasses the entire process of both splitting text into pieces and converting those pieces into numerical form.

The tokenization process is performed only when human language needs to be interpreted and understood; if Natural Language Processing (NLP/AI) is not being conducted, the text is not converted into matrices.

When we attempt tokenization, it is natural to think of three theoretical possibilities. When splitting sentences, we can break them down by words, by characters, or by word roots and suffixes.

If we do it at the Word-level, words like "Kitaplıktakiler" (the ones in the bookcase)—that is, words along with their suffixes—are processed as a whole, which naturally inflates the vocabulary size significantly. This can lead to Out-of-Vocabulary (OOV) bottlenecks for the model.

If we split at the Character-level (e.g., k-i-t-a-p...), the vocabulary becomes very small, but the semantic connection between words is lost and the input sequence becomes excessively long.

Therefore, the best approach is Subword-level splitting. It breaks words down into sub-components like roots and suffixes (kitap + lık + ta + kiler). This keeps the vocabulary size at an optimal level while solving the OOV problem by breaking rare words down into familiar subword pieces.