MLP

Whereas Attention allows tokens to retrieve information from one another, the MLP (Multi-Layer Perceptron) transforms each token’s hidden state before it is passed to the next Transformer layer.

Unlike Attention, the MLP processes each token independently. Once attention has gathered contextual information, every token can be transformed without looking at the other tokens.

Expanding the hidden state

The hidden state has a fixed size throughout the transformer. For example, in Qwen3.5-9B, every hidden state contains 4096 values.

Instead of transforming that vector directly, the MLP first expands it into a much larger internal representation, giving the model more capacity to learn complex transformations.

Once the computation has been performed, the representation is compressed back to its original size so it can continue through the transformer.

Modern LLMs generally perform three steps:

1. Expand the hidden state into a larger internal dimension
2. Apply a non-linear transformation to transform the expanded representation
3. Project the result back to the original hidden-state size

Like attention, the MLP does not replace the hidden state, but computes an update that is added back to the current representation. This refined hidden state is then passed to the next transformer layer, where the same process repeats.

What comes next?

Every generated token passes through every transformer layer, repeating the same sequence of Attention followed by the MLP.

To avoid unnecessary recomputation of Key and Value vectors for previously processed tokens, the next article explores the KV Cache mechanism.