Now that we understand that LLMs operate on patterns in text, let’s look at the most fundamental unit they work with. They don’t see words or letters as we do; they see ‘tokens’. Understanding this concept is the next step to mastering how to communicate with them effectively.
What Are Tokens?
Tokens are the smallest units of the text that an AI model processes. They’re not always whole words - they can be:
- Complete words (“hello”, “world”)
- Parts of words (“un”, “-ing”, “pre-”)
- Punctuation marks (“.”, “!”, “?”)
- Spaces and special characters
- Numbers or symbols
For example, the sentence “Hello, world!” might be tokenized as: [“Hello”, “,”, “world”, “!”] - that’s 4 tokens.
Why Are Tokens Integral to LLMs?
Tokens are crucial because:
- Computational Efficiency: Models work with numbers, not text. Each token maps to a numerical ID that the model can process mathematically.
- Vocabulary Management: Instead of trying to handle every possible word combination, models work with a fixed vocabulary of tokens (typically 50,000-100,000+ tokens).
- Handling Unknown Words: By breaking words into subword pieces, models can understand new or rare words by recognizing familiar components.
- Consistent Processing: Every piece of text gets converted to the same numerical format, allowing consistent mathematical operations.
The Tokenization Process
- Raw text → Tokenizer → Token IDs → Model processing
- Models learn relationships between these token IDs during training
- Output token IDs → Detokenizer → Human-readable text