Text
Start with
A prompt and the text so far
Generate
Predict → choose → append
Produce
A continuation, one token at a time.
The new token becomes part of the next prediction.
Shahzad Ali · Learning notes
Text, images & video.
How generative AI works across different media.
My visual notes on how models turn a prompt into text, an image or a moving scene, and the systems that make it possible.
Three common generation paths
A shared foundation.
Different processes.
These examples pair autoregressive text generation with diffusion-based image and video generation. Models can combine approaches, and the details vary.
Image
Start with
Noise, usually in a compressed image space
Generate
Refine → refine → decode
Produce
An image shaped by the prompt.
The text guides a learned denoising process.
Video
Start with
Noise representing a sequence of frames
Generate
Refine across space + time
Produce
A moving sequence shaped by the prompt.
The model also has to represent how things persist and move.
Training teaches the patterns. Generation uses those patterns and the current input. A plausible result still needs checking.
Text generation
Text builds a sequence
A language model processes the prompt and text generated so far, then scores possible next tokens. A decoding rule selects one. That token joins the context for the next prediction.
Tokens can be whole words, word pieces or punctuation. The Paris example shows the idea; its token IDs and probabilities illustrate a possible output. A high score still needs factual checking.
Tokens and textImage diffusion
An image emerges through refinement
A diffusion model learns to reverse a noise-adding process. During generation, it starts from noise and repeatedly refines a representation, guided by the prompt.
Latent diffusion performs this work in a compressed image space, then decodes it into pixels. The sampler and model determine the number of steps. Different image generators use different architectures and generation methods.
Generating with a trained modelVideo generation
Video adds time
A video model has to represent movement and continuity across a sequence: a person should remain recognisable as they walk, and objects should behave consistently from frame to frame.
OpenAI’s 2024 Sora report describes compressed visual data divided into spacetime patches and processed by a diffusion transformer. It also documents failures in physical interactions and long-range coherence. The visual pipeline alone establishes no audio capability.
TransformersTurning data into numbers
Embeddings are learned representations
An embedding represents an item as a vector: a list of numbers. The training objective shapes what relationships those numbers capture. Text and visual inputs can be represented in ways useful to the model.
The coloured clusters and three-column table are teaching sketches. Real representations usually have many more dimensions, and individual dimensions rarely carry simple labels such as “city” or “animal”. Similarity depends on the model and the task.
EmbeddingsInside the transformer
Attention combines available information
Attention calculates how to combine information from permitted positions. Feed-forward networks transform the resulting representations. Repeated blocks develop the numerical state used for the output.
The sketch shows one arrangement of residual connections and normalisation. Architectures vary. A causal text decoder can use the current and earlier positions; a bidirectional model or visual model follows its own attention pattern.
Attention and contextRetrieval-augmented generation · RAG
Retrieval brings extra material into the conversation
A retrieval system searches a collection for material relevant to a question. It supplies selected passages to the generator, giving the answer additional context. Embedding similarity is one way to find candidates.
For example: question → search company documents → retrieve relevant passages → add them to the prompt → generate an answer. At ordinary inference, this can change the available context while the trained weights stay fixed. Retrieval quality, document accuracy and answer checking still matter.
What context meansTraining and inference
Training changes the parameters
During training, an objective measures how the model is doing. For neural networks, backpropagation calculates gradients, and an optimiser uses them to update parameters. Inference uses the learned model on new input.
The number of examples, training updates and generation steps are different quantities. A model with billions of parameters can have a very different number of training updates. The effort required for inference depends on model size, output and usage.
How training worksData centres and compute
The model needs physical machinery
Large AI systems run calculations on accelerators such as GPUs. Memory holds parameters and working data; networks connect machines; storage keeps datasets and checkpoints. Electricity and cooling support the whole system.
Training large models and serving many requests both create demand for computing capacity. Smaller models can run on a single computer. Capacity, energy use and infrastructure depend on the workload and deployment.
What a GPU does
Sources behind the technical details
Sources for these notes.
Reviewed . The comparison covers common approaches. Research papers describe particular models; the Sora report and NVIDIA announcement are dated developer and manufacturer accounts.
- Primary research · First released
Language Models are Few-Shot Learners ↗
Brown and colleagues. Describes autoregressive language modelling, whole-model parameter counts and learning from context with fixed trained weights. Its results also document limitations.
- Primary research · First released
Attention Is All You Need ↗
Vaswani and colleagues. Introduces the transformer. Sections 3.1–3.5 describe attention, masking, feed-forward networks, residual connections, normalisation, embeddings and position.
- Primary research · First released
High-Resolution Image Synthesis with Latent Diffusion Models ↗
Rombach and colleagues. Describes diffusion in a learned compressed image representation, prompt conditioning and sampling trade-offs.
- Developer technical report · First released
Video generation models as world simulators ↗
OpenAI. Describes the 2024 Sora research system, spacetime patches and diffusion transformers, including observed limitations. This source concerns that visual research system.
- Primary research · First released
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks ↗
Lewis and colleagues. Combines a retriever with a generator. The original study also trains components; retrieval during ordinary use can supply new context without updating generator parameters.
- Manufacturer announcement · First released
Blackwell-powered DGX SuperPOD ↗
NVIDIA. A concrete example of GPUs, memory, interconnects, storage and cooling in a large AI system. Product specifications and performance claims are the manufacturer’s account.