Forward Propagation, visualized.

If you ship AI products, you will spend your career inside one acronym: the API call. A few lines of code go out, a blob of text comes back, and somewhere in the middle a neural network did something. That something is forward propagation — the act of moving data through a neural network in one direction, input to output, until a prediction falls out the end.

This is a walkthrough of what actually happens in there. By the end you'll have the mental models, vocabulary, and order-of-magnitude numbers to talk with ML engineers about latency, cost, and capability without nodding through words you don't own.

§ 00When your program calls an LLM

Before we zoom into the network, it helps to see where it sits inside the bigger picture. Here's what happens between the moment your program sends an LLM a request — some prompt like "write me a function that…" or "summarize this email" — and the moment a response streams back.

Two things are worth zooming in on: the prefill vs decode split inside the server, and what the model is doing when it starts feeding itself.

Prefill vs decode · two phases, two different rates

The forward pass isn't one step — it's two phases.

• Prefill — the opening shot. All 80 layers run once over your entire prompt, every input token processed in parallel. It fills the KV cache (the key/value vectors every attention layer will reuse) and produces one probability distribution over the first output token.
• Decode — the loop that follows. Sample a token, run another full forward pass on just that one token (reading the cache for everything before), stream it back, repeat.

A 500-token answer = 1 prefill + 500 decode passes.

From decode token #2 onward, the model is literally feeding itself — each pass's input is your prompt plus everything it has generated so far. That's the autoregressive in "autoregressive language model." The streaming typewriter effect in ChatGPT or Claude is nothing mystical — it's the decode loop made visible, each character a completed forward pass arriving over the wire the moment it's produced.

With that map in hand, let's zoom in. One API call runs many forward passes. One forward pass runs through 80 layers. One layer runs thousands of neurons. One neuron runs one small math operation. Each zoom down is the next thing to understand.

§ 01The assembly line

Think of a neural network as a multi-stage assembly line in a factory. Each stage — a "layer" — processes raw materials step by step, until a final product rolls off the line. Forward propagation is simply moving data down that line, in one direction, from input to output.

Every neural network — whether it's detecting spam, recognizing faces, or generating an essay — follows the same four-stage anatomy. The stages vary in size and sophistication, but the structure is universal.

STAGE 01

Input layer · Raw materials

The starting point. An email's text. An image's pixels. A customer profile's columns. Whatever you're asking the model to reason about, in its rawest numerical form. For an LLM, this is the embedded tokens we just met.

Flour, eggs, sugar arriving at the bakery — unprocessed, unmixed, full of potential.

STAGE 02

Hidden layers · Assembly stations

The intermediate steps where data is transformed. Early layers catch simple patterns — edges, keywords. Later layers compose those into richer ones: an ear, then a face, then cat. "Hidden" just means you don't see their outputs directly — they're intermediate work.

Mix the batter. Shape the cake. Frost and decorate. Each station adds one meaningful transformation.

STAGE 03

Neurons · The workers

Each neuron is a tiny voter inside a layer. It takes its inputs, multiplies each by a weight ("how much should I care about this signal?"), adds a small bias, runs the result through an activation function, and decides: fire, or stay quiet. We'll zoom in next — they're the protagonists.

A quality-check station where workers inspect parts and vote on whether something is worth passing along.

STAGE 04

Output layer · The verdict

The last layer produces the prediction. Spam or not spam. An estimated price. An identified object. The next word to say. Whatever question the model was built to answer — usually with a confidence score attached.

The finished cake, boxed, ribboned, ready for delivery.

§ 02Inside one neuron

The input layer is really just data sitting there — no workers, no calculations, just numbers waiting. Everything interesting starts at the first hidden layer, where the first wave of neurons springs into action. Understand one neuron and you understand the whole machine.

Two things to internalize.

Weights are what "learning" produces. When a model is trained, the weights are the knobs being tuned. Billions of tiny dials get nudged, epoch by epoch, until the network gets good at its task. Everything a model "knows" is encoded in those numbers. When someone says "the weights of GPT-4 aren't public," they mean: you can't download the dials.

The activation is what makes depth worth having. Without that non-linear kink at each neuron, stacking a hundred layers would be mathematically equivalent to one giant linear equation — you could only learn straight lines. The activation introduces the small non-linearity that, compounded across many layers, lets the network model arbitrarily complex patterns. It's the reason deep learning is deep.

What do the layers actually do?

Once you stack 80 of these layers and train them on the internet, specific layers end up specializing in specific things. Roughly — but less tidily than vision models, where early layers clearly detect edges and late layers clearly detect objects. In language models, interpretability researchers find bands, not hard boundaries.

Three caveats keep the picture honest:

• Bands overlap. The same function often appears across many layers.
• Models specialize differently. These are tendencies, not laws.
• Specialization is inferred, not labelled. Researchers probe each layer's outputs on tasks like part-of-speech tagging or entity recognition and measure how well they predict.

The field that maps this out is mechanistic interpretability; Anthropic's Transformer Circuits is the canonical starting point.

The usable intuition for product work: depth enables composition. You don't need to know exactly which layer holds the "sentiment" circuit to reason about what models can and can't do — but when an engineer says "we're fine-tuning the top few layers," you now know they're touching task-format and output machinery, not rewiring how the model reads syntax.

§ 03How big are we actually talking?

An assembly line sounds manageable. The reality is that modern networks have so many workers and stations that the numbers stop feeling like numbers and start feeling like astronomy. Here's the calibration your intuition needs.

Parameters is what ML folks call weights + biases: the total count of tunable dials in the model. It's the most-quoted stat about a model, and it's the right one for order-of-magnitude intuition.

Anatomy of a modern LLM

Let's look at one in detail. Llama 3 70B, because its architecture is public and it's representative of the current state of the art.

Where all those gigabytes come from

Model size on disk is just parameters × bytes per parameter. Each parameter is a number, and that number needs to be stored at some precision.

Quantization is the art of representing each parameter with fewer bits. A weight that was 0.7234156 in training might become just 0.72 — or even 0.7 — at inference. The model gets dumber, often only slightly, and memory drops 2×, 4×, sometimes 8×. This is why you see people running "7B" or "13B" models on a laptop: the base model is small and quantized.

Are the layers "fixed"? Architecture vs weights

Natural follow-up once you see numbers like these: are the layers fixed, or do they shift around between users, prompts, tokens? The answer needs two distinctions.

• The architecture is fixed. Once a model is defined, the structure is frozen — 80 layers, 8,192-dim hidden state, 64 attention heads per layer, in a specific arrangement. Every forward pass for every user, every prompt, every token runs through the same machinery in the same order. You can't skip a layer or add one at inference time.
• The weights are fixed after training. The billions of numbers inside those layers freeze the moment training ends. When you call the Anthropic or OpenAI API today, every request hits the same weights. That's what makes "Llama 3 70B" or "Claude Sonnet 4.6" a specific shippable bundle — deterministic, reproducible.
• Only the input changes. Your tokens, and therefore the activations flowing through the network. The plumbing is identical; what flows through it is different. The apparent randomness in LLM outputs comes from the sampler (temperature, top-p) picking tokens from the probability distribution — not from the network running differently each time.

Who designs the architecture? Humans — the ML research team at the lab. The architecture file is usually a few hundred lines of code. Meta's Llama 3 is a good public reference — the entire architecture is ~300 lines of PyTorch: meta-llama/llama3/llama/model.py. If you open it, three lines make the "stacked layers" picture concrete:

• n_layers (line 21, inside ModelArgs) — the depth parameter. Defaults to 32 in the file, but set to 80 when loading the 70B variant.
• class TransformerBlock (line 251) — the definition of one layer: attention + feedforward + normalization, wrapped together. This is "Stage 02" from §01 in code form.
• for layer_id in range(params.n_layers): self.layers.append(TransformerBlock(...)) (around line 288) — the stacking. Literally: "put 80 of these in a list, in order." That loop is the assembly line.

What makes a model "smart" isn't the architecture; it's what gets poured into it during training. We'll get to that direction of the assembly line in §06.

§ 04The latent space

Here is the single most important concept you don't yet have a picture for. Bring this one to whiteboards and watch engineers nod — it's the abstract "workspace" the model reasons inside.

The output of every hidden layer is a list of numbers — a vector. In Llama 3 70B, that list is 8,192 numbers long. You can't picture an 8,192-dimensional space, but you can picture the idea: each input the model has ever seen ends up as a point in that space. Similar things — "cat" and "dog," "joy" and "hope" — land near each other. Dissimilar things land far apart. This is the latent space (sometimes called embedding space or representation space).

The famous result that first made this click: in a well-trained language model, you can do arithmetic on meanings. Take the vector for "king," subtract "man," add "woman" — and you land very close to "queen." The model has organized its internal space so that abstract relationships (gender, royalty, tense, plurality) correspond to consistent directions.

Where it fits on the assembly line

Latent space isn't a separate stage — it is what the hidden layers are producing. Each layer's output is a point in a (usually high-dimensional) latent space. As data moves through the network, its location in latent space shifts: early layers organize by surface features (letter patterns, edges), deep layers by abstract meaning (sentiment, intent, topic).

Same space family, different depths:

• "The embedding" — output of the very first transformation; the input's initial coordinates.
• "Hidden state" — the latent-space position at some intermediate layer.
• "Final representation" — near the output end.

§ 05End-to-end: a spam classifier

The simplest possible worked example — a tiny network deciding if one email is spam. Same anatomy as a trillion-parameter LLM, just smaller.

Reading left to right: a raw email enters on the left. The first hidden layer picks out suspicious words in isolation — free, click, winner, now. The next layer notices these keywords aren't just present individually; they're clustering together in ways that matter. The output layer weighs it all and commits: spam, 90% confidence.

The whole journey is forward propagation. One direction, one pass, one answer. The "learning" already happened during training — thousands of example emails, weights slowly settling into positions that make this particular pipeline accurate. Now, at inference time, the model is just running what it knows.

Scale the picture up. Replace the four stages with 80 transformer blocks. Replace the tiny vocabulary of suspicious words with a token embedding table. Replace the binary "spam / not spam" verdict with a probability over 128,000 possible next tokens. Run the whole thing once per token you want to generate. That's an LLM.

§ 06The other direction · backpropagation

Forward propagation is the model running. Backpropagation is how the model learned to run — the reverse direction through the same network, used only during training, never at inference time. Every weight you now find frozen inside Llama 3 or Claude got there because of this loop.

The training loop

For each training example — a chunk of text from the internet, a book, source code, a curated conversation:

• Forward pass — run the input through the network the way we've been describing. Out falls a prediction.
• Loss — compare the prediction to the correct answer (the actual next token, the actual label). The difference is a single number measuring how wrong the model was.
• Backward pass — walk from the output back through every layer, computing, for each of the billions of weights, which direction would have made the loss smaller. That direction is called the gradient.
• Update — nudge every weight a tiny amount in that direction.
• Repeat — trillions of times.

Over training, the weights drift from random noise into a configuration where the forward pass produces useful answers. That drift is where "learning" happens. By the end, everything the model "knows" — grammar, facts, reasoning patterns, style — is encoded in those billions of numbers.

Why "back"-propagation?

Because the gradient flows backward through the layers, exactly mirroring the forward pass. The error at the output tells you what the last layer should have done differently. That signal tells the second-to-last layer what it should have done. And so on, back to the first hidden layer. Each layer's contribution is computed using the chain rule of calculus — but the intuition is simpler than the math: same network, signal flowing the other way.

Pre-training, fine-tuning, inference — same machinery, different scales

• Pre-training — trillions of tokens, weeks on thousands of GPUs. Produces a general-purpose model. Millions of dollars per frontier model.
• Fine-tuning / RLHF — a much smaller curated dataset nudges the already-trained weights toward specific behaviors ("follow instructions," "avoid harm," "match this style"). Same forward + backward + update loop; days and thousands of dollars, not weeks and millions.
• Inference — forward pass only. Cheap. What your API calls hit.

§ 07Why this matters at the PM level

Trade-off

More layers = richer patterns, harder debugging. Depth buys capability but costs you interpretability and training stability. Every architectural decision is a product decision — capacity vs. cost, flexibility vs. controllability.

Latency

Each layer adds compute time. A forward pass through 80 layers is 80 sequential matrix multiplications per token. For streaming use cases this surfaces as time-to-first-token and tokens-per-second — the two metrics your users will actually feel.

Interpretability

Early layers are legible; later ones are abstract. You can often tell what a first layer detects. By layer forty, features are compositions of compositions. Explainability tooling exists for a reason — mechanistic interpretability is a whole field (Anthropic does a lot of it).

Cost

Parameter count drives everything. Memory, inference cost, training compute, deployment options. Scaling from a 7B to a 70B model is a 10× jump in every dimension — and not always a 10× jump in quality. This is the math behind "why not just use GPT-4 for everything."

Context

The latent space is where your product lives or dies. RAG, semantic search, personalization — all latent-space engineering. The model's representation of your domain is the substrate your features are built on.

TL;DR

Forward propagation is a step-by-step recipe for turning raw data into a decision. Inside an LLM call it runs once per output token.

1. Input enters as numbers. For an LLM, that's tokens from a tokenizer, then vectors from an embedding table.
2. Hidden layers transform. Each neuron is a weighted-sum-plus-activation deciding whether to pass a signal forward. Stacked deep, they compose simple patterns into abstract meaning.
3. Latent space is the workspace. At every layer the data occupies a point in a high-dimensional space — progressively reorganized to highlight what matters for the task.
4. The output layer commits. A class, a score, a probability distribution over the next token.

The "learning" happened during training, when billions of weights were tuned against examples. At inference time the model is just applying what it already knows — running the assembly line it's been taught to run. Understanding this gets you 80% of the way in any ML conversation; the rest is scale, architecture, and taste.