Shahzad Ali

Shahzad Ali · Learning notes

Inside a transformer.

From text, to numbers, to the next token.

My notes on what happens inside a GPT-style language model. A sequence of calculations turns the available context into a prediction of what comes next.

An imagined transformer built from banks of adjustable knobs. Text enters on the left, passes through embedding and transformer layers, then next-token scores leave on the right. The repeat loop is explained below.
My AI-assisted visual sketch of a transformer. Each knob represents a learned number. View full size ↗

Follow the flow

A response grows
one token at a time.

A simplified decoder-only language model. The example is scripted to show the loop; token boundaries vary between models.

Context available now

The cat sat on the

  1. 01Tokens + positionRepresent the input as numbers in order.
  2. 02Attention + feed-forwardPass through the trained layers.
  3. 03Next-token scoresForm a distribution over the vocabulary.
  4. 04Choose + appendThe selected token joins the context.

Use the expanded context for the next prediction.

The trained weights stay fixed.

Example continuation: “ mat.” Two illustrated steps: mat, then . Generation can then finish with an end-of-sequence token.

  1. Tokenisation

    Text becomes token IDs

    A tokeniser splits text into pieces and assigns each piece an ID. A piece can be a word, part of a word, punctuation or a space-related sequence.

    The word boxes in the picture make the idea visible. A real tokeniser may split the same sentence differently, depending on its vocabulary.

    Tokenisation
    Brown and colleagues

  2. Embeddings and position

    IDs become useful numbers

    An embedding maps each token ID to a vector: a list of numbers. Position information lets the network account for order.

    “The dog chased the cat” and “The cat chased the dog” contain the same words in different roles. Order changes the task the model has to solve.

    Embeddings
    Vaswani and colleagues

  3. Self-attention

    Each position gathers context

    Attention lets each position draw on useful information from the available tokens. It calculates how strongly to combine information from each permitted position.

    Queries and keys calculate the matching scores; values carry the information being combined. A causal decoder uses the current and earlier positions. These attention scores change with the input; the trained parameters stay fixed during ordinary inference.

    Attention
    Vaswani and colleagues

  4. Feed-forward layers

    The representations change again

    After attention combines information across tokens, a feed-forward network transforms the numbers at each position. Repeating these blocks develops the representation used for prediction.

    Residual connections carry information around a block, and normalisation helps control numerical scale. The original transformer combines these ingredients; later architectures alter the details.

    Transformer blocks
    Vaswani and colleagues

  5. The output

    Scores become a next-token choice

    The final representation produces a score for each token in the vocabulary. Softmax turns the scores into a probability distribution. A decoding rule chooses a token from it.

    A system might select the highest-probability token or sample from several plausible ones. This distribution describes a continuation of the text; factual accuracy requires separate checking.

    Probability
    Vaswani and colleaguesBrown and colleagues

  6. Repeat the loop

    The chosen token joins the context

    After a token is selected, it becomes part of the sequence used to predict the next one. The response grows until a stopping condition is reached.

    Implementations often reuse stored key and value vectors from earlier tokens, reducing repeated calculation. Context capacity is limited, and generation speed depends on the model and the hardware.

    Context
    Brown and colleaguesKwon and colleagues

  7. Training versus inference

    Training sets the knobs

    Training measures prediction error and uses it to adjust parameters over many examples. Ordinary inference uses those learned parameters to process the current context.

    During training, a known sequence supplies targets for many positions at once while causal masking preserves the order of prediction. During generation, each selected token feeds the next step.

    How training works
    Brown and colleaguesVaswani and colleagues

  8. The wider family

    One design, several uses

    The original transformer used an encoder and a decoder for translation. BERT uses a bidirectional encoder. GPT-style generators use a causal decoder.

    Architecture and training objectives shape what the model can do. These calculations alone leave questions about consciousness and subjective experience open.

    The transformer family
    Devlin and colleaguesVaswani and colleaguesBrown and colleagues

Sources behind the technical details

Sources for these notes.

Reviewed . This is a simplified account of an autoregressive decoder-only language model. Exact architecture, tokeniser and decoding choices vary.

  1. Primary research · First released

    Attention Is All You Need ↗

    Vaswani and colleagues. Introduces the transformer. Sections 3.1–3.5 describe attention, masking, feed-forward networks, residual connections, normalisation, embeddings and position.

  2. Primary research · First released

    Language Models are Few-Shot Learners ↗

    Brown and colleagues. Describes autoregressive language modelling, whole-model parameter counts and learning from context with fixed trained weights. Its results also document limitations.

  3. Primary research · First released

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding ↗

    Devlin and colleagues. Masked-language training supplies targets from the text itself. BERT uses bidirectional context, showing how transformer designs can serve different tasks.

  4. Primary research · First released

    Efficient Memory Management for Large Language Model Serving with PagedAttention ↗

    Kwon and colleagues. Explains key–value caching during autoregressive generation and how the vLLM system manages that memory.

← Back to my computing notes