Embeddings

The tokenizer converts a prompt into a list of token IDs. These IDs are simply integers; they don’t carry any semantic information. The first layer of a model, called the embedding layer, retrieves the embedding vector corresponding to each token ID.

The tokenizer is outside the model, whereas the embedding layer is inside the model.

What is an embedding?

A token ID is an integer; it doesn’t contain any semantic meaning by itself. An embedding is a high-dimensional vector that captures semantic relationships between tokens. There is one embedding vector for every token in the vocabulary. These vectors are learned during training and stored inside the model’s weights.

In this high-dimensional space, embeddings for tokens “cat”, “dog”, “kitten” are close to each other, whereas embeddings for “cat”, “banana” are far from each other as they have no similar meaning.

The length of every embedding vector is called d_model. For Qwen3.5-9B, d_model=4096. This means every token is represented by an embedding containing 4096 numbers.

Embedding lookup

Because the tokenizer assigns each token an ID, the embedding layer can directly retrieve the corresponding row from the embedding matrix. This is why the tokenizer used during inference must match the one used during training.

The embedding layer does not compute the embedding from a token ID, it only retrieves it from the model’s weights.

Example

The following Python script displays information about the model’s embedding matrix and the embedding of a specific token ID.

from transformers import AutoModelForCausalLM

model_id = "Qwen/Qwen3.5-9B"
model = AutoModelForCausalLM.from_pretrained(model_id)

# Dimensions of the embedding tensor
print(model.model.embed_tokens.weight.shape)

# Dimensions and extract of the embedding for token 3710
embedding = model.model.embed_tokens.weight[3710]
print(embedding)
print(f"Embedding shape : {embedding.shape}")

The output is similar to the following:

torch.Size([248320, 4096])
tensor([ 0.0078,  0.0003, -0.0106,  ...,  0.0075,  0.0073, -0.0096],
       dtype=torch.bfloat16, grad_fn=<SelectBackward0>)
Embedding shape : torch.Size([4096])

From this result, we can see that the size of the embedding tensor is [248320, 4096], which means the model stores one 4096-dimensional vector for each of the 248320 tokens in its vocabulary.

What’s next?

After the embedding layer, each token is represented by its embedding vector. These vectors then enter the first Transformer layer, where attention and MLP progressively refine them into a context-aware hidden state.