How does it learn?
Labelled examples, structure in data, prediction targets taken from the data, or rewards from actions.
Shahzad Ali · Learning notes
The AI machinery room.
How the different approaches fit together.
My visual notes on the machinery behind AI: how a system learns, how it is built and what it produces.
The map I keep in mind
Three questions.
One system.
AIThe broad field
Machine learningLearning from data or experience
Deep learningLearning through many neural-network layers
How is it built?
The arrangement of the computation. A transformer is one kind of neural-network architecture.
What does it do?
A model might classify an image, estimate a number, choose an action or generate new content.
Put them together · A GPT-style language model
Predict text during trainingTransformer architectureGenerate text when used
Later training can add demonstrations and feedback. Several labels can describe the same system.Supervised learning
Learning from examples
Give a model examples with answers: emails labelled “spam” or “inbox”, for instance. Training adjusts the model to improve its predictions on those examples.
The useful question is how well it handles new examples. Labels, data quality and the problem being solved all matter.
Neural networksUnsupervised learning
Finding structure
Give a model data without an answer label for each example. It may group similar items or find a simpler way to represent them.
For example, a shop could group customers by purchasing patterns. A cluster shows similarity under the chosen method; explaining its cause needs further investigation.
Patterns and representationsSelf-supervised learning
The data supplies the target
Text can supply its own training questions. Hide a token and predict it, or use earlier text to predict the following token. That creates many examples without someone labelling every answer.
BERT learns from masked text. A GPT-style model learns next-token prediction. Both can learn useful representations before additional training for specific uses.
PretrainingReinforcement learning
Actions have consequences
An agent takes actions, receives rewards and learns a policy: a way to choose what to do. The aim is a higher total reward over time.
In the DQN work, a deep neural network learned to play Atari games from screen pixels and rewards. This connects reinforcement learning directly with deep learning. The reward defines what success means to the system.
Reinforcement learningDeep learning
Layers build representations
A deep neural network transforms its input through many layers. Each stage creates numerical representations that later stages can use.
The “knobs” are learned parameters. The picture’s steps from patterns to reasoning are a visual simplification. Real networks vary in depth and structure, and individual layers rarely have one neat human label.
Inside a neural networkNetwork architectures
Different arrangements for different data
CNNs reuse filters across local regions, such as parts of an image. RNNs carry a state through a sequence. Transformers use attention to combine information. GNNs pass information along the links in a graph.
An image, a sentence and a molecule have different structures. These designs offer ways to use that structure. Many systems combine several components.
The transformerGenerative models
Creating a new example
An autoregressive language model builds text token by token. A GAN trains a generator alongside a discriminator. A standard diffusion model learns to reverse added noise, then uses repeated denoising to generate a sample.
These approaches learn from training examples and can produce new text, images or other data. A generated result still needs checking for the task you want it to perform.
Generating with a modelFrom data to a useful system
Training and using are different stages
Training changes the model’s parameters. Inference uses the trained model on an input. A typical chat supplies fresh context while the parameters remain fixed.
The application adds its own decisions: which data to use, what success means and how to check results. Fluent language and attractive images alone cannot establish accuracy.
Training and inference
Sources behind the technical details
Sources for these notes.
Reviewed . The machinery is an illustration. The papers and textbook explain the underlying methods.
- Authors’ textbook · Published 2016
Deep Learning · Chapters 1, 9 and 10 ↗
Goodfellow, Bengio and Courville. The introduction maps AI and deep learning. Chapters 9 and 10 explain convolutional filters and recurrent networks.
- Primary research · First released
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding ↗
Devlin and colleagues. Masked-language training supplies targets from the text itself. BERT uses bidirectional context, showing how transformer designs can serve different tasks.
- Primary research · First released
Human-level control through deep reinforcement learning ↗
Mnih and colleagues. The DQN system combines a deep neural network with reinforcement learning on Atari games.
- Primary research · First released
Attention Is All You Need ↗
Vaswani and colleagues. Introduces the transformer. Sections 3.1–3.5 describe attention, masking, feed-forward networks, residual connections, normalisation, embeddings and position.
- Primary research · First released
Neural Message Passing for Quantum Chemistry ↗
Gilmer and colleagues. Describes learning on molecular graphs by passing and aggregating information between connected nodes.
- Primary research · First released
Language Models are Few-Shot Learners ↗
Brown and colleagues. Describes autoregressive language modelling, whole-model parameter counts and learning from context with fixed trained weights. Its results also document limitations.
- Primary research · First released
Generative Adversarial Networks ↗
Goodfellow and colleagues. Introduces joint training of a generator and a discriminator.
- Primary research · First released
Denoising Diffusion Probabilistic Models ↗
Ho and colleagues. Defines a noise-adding process and a learned reverse process for generating samples.