Transformer
A transformer is the core of modern LLMs. Once the prompt has been converted into embeddings, each embedding serves as the initial hidden state of a token and passes through a sequence of transformer layers. These hidden states are progressively refined before they reach the LM Head.
A sequence of layers
Modern LLMs contain tens of transformer layers. For example:
- Qwen/Qwen3.5-9B contains 32 transformer layers
- openai/gpt-oss-120B contains 36 layers
- swiss-ai/Apertus-70B-2509 contains 90 layers
Each layer has learned different weights during training. As a result, different layers often learned to extract different kinds of information from the input hidden state.
The hidden state
Every token flowing through the transformer is represented by a vector called its hidden state. Each layer receives the hidden states produced by the previous one, refines them, and passes them to the next layer.
The embedding layer produces the first hidden state of every token before the transformer layers begin to refine it.
What happens inside one layer?
Every transformer layer performs two operations:
- Attention enriches the hidden state using information from previous tokens in the sequence
- MLP (Multi-Layer Perceptron) enriches that hidden state using the knowledge learned during training
Both operations produce an update that is added back to the hidden state before the next operation begins.
As the hidden state passes through successive transformer layers, it gradually incorporates more contextual information from the rest of the sentence and from the model’s internal knowledge. Each layer refines the hidden state a bit more.
What comes next?
A transformer layer consists of 2 operations: attention and MLP. The next 2 articles explore these operations in more detail.