Tokenizer

Humans write text, LLMs process numbers. Before a model can process a prompt, a tokenizer converts it into a sequence of token IDs.

A tokenizer performs two steps:

  • split the text into tokens
  • convert each token into a unique integer ID

What is a token?

A token may represent a whole word, part of a word, a punctuation character, or even a space. The way a prompt is split into tokens depends on the tokenizer used.

For example, “What is Kubernetes?” may be split into [“What”, “Ġis”, “ĠKubernetes”, “?”], whereas “unbelievable” may be split into [“un”, “bel”, “ievable”].

Tokenizers do not simply split on spaces. They use algorithms such as Byte Pair Encoding (BPE) or SentencePiece to efficiently represent text.

The Ġ prefix indicates that the token is preceded by a space. Other tokenizer families may use different markers.

Converting tokens into IDs

The next step is to convert each token into an ID using the model’s vocabulary, which maps every possible token to a unique integer.

Where does the tokenizer come from?

The tokenizer is usually stored in a tokenizer.json file alongside the model weights. The screenshot below shows this file for the Qwen3.5 9B model.

List of files for QWen 3.5 9B on HuggingFace
List of files for QWen 3.5 9B on HuggingFace

This file contains the tokenizer configuration, including the vocabulary used to map tokens to IDs.

$ grep 'What' tokenizer.json
      "ĠWhat": 3437,
      "What": 3710,          <- First token of our sample prompt
      "ĠWhats": 28785,
      ...

A couple of things to consider:

  • The tokenizer used during inference must be the same as the one used during the training

  • Different models in the same family (e.g. Qwen) often share the same tokenizer

  • For backward compatibility, many repositories also include vocab.json. Its vocabulary is already contained inside tokenizer.json

Examples

The following python script shows how a simple prompt is split into tokens and associated with token ids.

from transformers import AutoTokenizer

model_id = "HUGGINGFACE_MODEL_IDENTIFIER"
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Text to tokenize
text = "What is Kubernetes?"

# Get token IDs from text
token_ids = tokenizer.encode(text, add_special_tokens=False)
tokens = tokenizer.convert_ids_to_tokens(token_ids)

print("Vocabulary size:", tokenizer.vocab_size)
print("Tokens:", tokens)
print("Token IDs:", token_ids)

The following examples use 2 different models to show that the resulting token IDs are different.

  • Qwen3.5-9B
Vocabulary size: 248044
Tokens: ['What', 'Ġis', 'ĠKubernetes', '?']
Token IDs: [3710, 369, 64135, 30]
  • Gemma-4-12B-it
Vocabulary size: 262144
Tokens: ['What', '▁is', '▁Kubernetes', '?']
Token IDs: [3689, 563, 63922, 236881]

what’s next

Tokens IDs are simply identifiers. They do not contain any semantic information. The model first converts them into embeddings, which are high dimension vectors containing semantic meaning.