Skip to main content
MICHIEL HORSTMAN
BOOK A CALL ↗

Transformer Systems From Scratch: Architecture, Training, and Production Deployment

TransformersLLMDeep LearningProduction AIModel Architecture
Transformer Systems From Scratch: Architecture, Training, and Production Deployment

Transformer Systems From Scratch: Architecture, Training, and Production Deployment

Transformer architecture is the foundation behind every major language model in production today, from GPT-4 to Claude to Gemini. Understanding it at the system level is no longer optional for engineers building AI products. Read time: 28 min.

Most resources on transformers fall into two camps. The academic ones drown you in attention formulas and leave you unable to ship anything. The applied ones hand you a model card and call it a day, leaving you wondering what's actually happening under the hood.

This article is the bridge. We'll implement the core architecture from scratch, then trace every component through to production deployment. By the end, you'll understand not just what a transformer does, but why every design decision was made and how it affects your production system.


What a Transformer Actually Does (In One Sentence)

A transformer converts a sequence of tokens into a sequence of predictions by letting every token "attend" to every other token, weighted by relevance.

That's it. The rest is engineering: making it fast enough, stable enough, and scalable enough to train on trillions of tokens and serve billions of requests.

Everything that follows is a practical consequence of that one sentence. The architecture exists because we needed a way to process sequences in parallel (RNNs process one token at a time). The attention mechanism exists because we needed a way to weigh token relationships dynamically (fixed context windows don't capture long-range dependencies). The training pipeline exists because we needed a way to optimize 100B+ parameters without losing numerical stability.


The Attention Mechanism: Implemented and Optimized

Attention is the heart of the transformer. Everything else is supporting infrastructure. Here's the minimal implementation.

import torch
import torch.nn as nn
import torch.nn.functional as F
import math

class ScaledDotProductAttention(nn.Module):
    """The core attention mechanism — every token looks at every other token."""

    def __init__(self, d_k: int):
        super().__init__()
        self.d_k = d_k

    def forward(self, Q: torch.Tensor, K: torch.Tensor,
                V: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:
        # Q, K, V shapes: (batch, heads, seq_len, d_k)
        scores = torch.matmul(Q, K.transpose(-2, -1))
        scores = scores / math.sqrt(self.d_k)  # Scale to prevent softmax saturation

        if mask is not None:
            scores = scores.masked_fill(mask == 0, float("-inf"))

        weights = F.softmax(scores, dim=-1)
        return torch.matmul(weights, V)

The scaling by sqrt(d_k) is not a minor implementation detail. Without it, when d_k grows large, the dot products grow large too, pushing softmax into regions where gradients vanish. The model stops learning. This single line — scores / math.sqrt(self.d_k) — is what makes training deep transformers possible.

Multi-Head Attention: Why One Head Is Never Enough

A single attention head learns one set of relationships. Multi-head attention learns multiple relationship types simultaneously: syntactic relationships, semantic relationships, positional relationships, and whatever else the data reveals.

class MultiHeadAttention(nn.Module):
    def __init__(self, d_model: int, n_heads: int):
        super().__init__()
        assert d_model % n_heads == 0, "d_model must be divisible by n_heads"

        self.n_heads = n_heads
        self.d_k = d_model // n_heads

        self.W_q = nn.Linear(d_model, d_model, bias=False)
        self.W_k = nn.Linear(d_model, d_model, bias=False)
        self.W_v = nn.Linear(d_model, d_model, bias=False)
        self.W_o = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:
        batch_size, seq_len, _ = x.shape

        # Project and reshape into heads
        Q = self.W_q(x).view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        K = self.W_k(x).view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        V = self.W_v(x).view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)

        # Apply attention
        attention = ScaledDotProductAttention(self.d_k)
        out = attention(Q, K, V, mask)

        # Concatenate heads and project
        out = out.transpose(1, 2).contiguous().view(batch_size, seq_len, -1)
        return self.W_o(out)

Each of the W_q, W_k, W_v projections is a learned linear transformation. The model learns what each head should specialize in. In practice, some heads learn to track subject-verb agreement, others learn positional patterns, and a few become "null heads" that attend uniformly — a discovered robustness mechanism.

Flash Attention: The Production Reality

The attention mechanism I implemented above is the textbook version. Production systems don't use it. They use Flash Attention, which combines attention computation with memory optimization.

Standard attention materializes the full attention matrix — for a 4096-token sequence with 32 heads, that's 4096 x 4096 x 32 = ~537 million values in GPU memory. Flash Attention tiles the computation across the SRAM of the GPU, never materializing the full matrix at once.

# Production systems call this instead:
from flash_attn import flash_attn_func

# Same semantics, 3-10x faster, 10-20x less memory
out = flash_attn_func(Q, K, V, dropout_p=0.0, causal=True)
# Q, K, V shapes: (batch, seq_len, n_heads, d_k) — note different layout!

The API difference (source-first vs batch-first dimension ordering) has tripped up more than one team migrating from textbook to production. The causal=True flag masks future tokens — essential for autoregressive generation.


The Full Transformer Block: Putting It Together

A transformer is a stack of identical blocks, each containing multi-head attention and a feed-forward network, wrapped in residual connections and layer normalization.

class TransformerBlock(nn.Module):
    def __init__(self, d_model: int, n_heads: int, d_ff: int, dropout: float = 0.1):
        super().__init__()
        self.attention = MultiHeadAttention(d_model, n_heads)
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),  # GELU, not ReLU — smoother gradient flow
            nn.Linear(d_ff, d_model),
            nn.Dropout(dropout),
        )
        self.norm1 = nn.RMSNorm(d_model, eps=1e-5)
        self.norm2 = nn.RMSNorm(d_model, eps=1e-5)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x: torch.Tensor, mask: torch.Tensor = None) -> torch.Tensor:
        # Pre-norm architecture (standard since GPT-2)
        x = x + self.dropout(self.attention(self.norm1(x), mask))
        x = x + self.dropout(self.ffn(self.norm2(x)))
        return x

Pre-Norm vs Post-Norm: Why the Industry Switched

The original transformer paper placed layer normalization after the residual connection (post-norm). Every modern implementation places it before (pre-norm). The reason: pre-norm stabilizes training by keeping the residual stream's gradient scale consistent.

# Post-norm (original, unstable for deep models)
def post_norm_block(x, attn, ffn, norm):
    x = norm(x + attn(x))  # Gradient flows through norm first
    x = norm(x + ffn(x))   # Norm acts on the sum, not the branch
    return x

# Pre-norm (modern, stable for 100+ layer models)
def pre_norm_block(x, attn, ffn, norm):
    x = x + attn(norm(x))  # Gradient flows directly to x
    x = x + ffn(norm(x))   # Residual path is identity
    return x

With pre-norm, the residual stream maintains a direct gradient path through all layers. With post-norm, each norm layer compresses the gradient, making it harder for information to flow back through deep networks.

FFN: The Silent Majority of Compute

The feed-forward network accounts for roughly two-thirds of a transformer's FLOPs, even though it's structured as two linear layers. That's because the hidden dimension d_ff is typically 4x the model dimension d_model.

# FFN input: (batch, seq_len, d_model) where d_model = 4096
# FFN hidden: (batch, seq_len, d_ff) where d_ff = 16384
# FFN output: (batch, seq_len, d_model) where d_model = 4096

The GELU activation function is chosen over ReLU for a specific reason: it has smooth gradient everywhere, which improves training stability. The approximation used in practice is:

def gelu_approx(x):
    return 0.5 * x * (1 + torch.tanh(math.sqrt(2/math.pi) * (x + 0.044715 * x**3)))

This matters because tensor core units on modern GPUs compute this approximation efficiently using fused operations. The exact GELU is slower and provides negligible quality benefit.


Training at Scale: The Pipeline That Actually Trains

Training a transformer from scratch requires more than just the model. The training infrastructure is where most projects fail. Here's the production training pipeline.

Data Loading: The Hidden Bottleneck

Your GPU is faster than your data pipeline. Even with high-end SSDs and fast networking, most training runs spend 20-40% of their time waiting for data. The fix: shard aggressively, prefetch late, and store in a format designed for random access.

import numpy as np
from torch.utils.data import Dataset, DataLoader

class ShardedDataset(Dataset):
    """Distributed dataset sharded across training nodes."""

    def __init__(self, shard_dir: str, global_rank: int, world_size: int,
                 sequence_length: int = 4096):
        shards = sorted(glob.glob(f"{shard_dir}/*.npy"))
        # Each rank loads only its shards
        self.shards = [s for i, s in enumerate(shards)
                       if i % world_size == global_rank]
        self.sequence_length = sequence_length
        self._load_shard(0)

    def _load_shard(self, index: int):
        self.data = np.memmap(self.shards[index], dtype=np.uint16, mode='r')

    def __getitem__(self, idx: int) -> dict:
        start = idx * self.sequence_length
        tokens = torch.from_numpy(
            self.data[start:start + self.sequence_length + 1].astype(np.int64)
        )
        return {"input_ids": tokens[:-1], "labels": tokens[1:]}

The uint16 dtype is deliberate. Token IDs fit within 16 bits (our vocabularies are under 65K). Using float32 would waste 2x memory. Every optimization at this level compounds across trillions of tokens.

Mixed Precision Training

Modern GPUs train 2-3x faster with mixed precision (FP16/BF16) while maintaining model quality. The key insight: not every operation needs FP32 precision. Weight updates need it (gradient accumulation), but forward passes can run in half precision.

from torch.cuda.amp import autocast, GradScaler

scaler = GradScaler()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4)

for batch in dataloader:
    with autocast(dtype=torch.bfloat16):  # BF16 for forward pass
        outputs = model(batch["input_ids"])
        loss = F.cross_entropy(
            outputs.view(-1, vocab_size),
            batch["labels"].view(-1)
        )

    scaler.scale(loss).backward()  # Backprop in FP32
    scaler.step(optimizer)
    scaler.update()

BF16 (bfloat16) is preferred over FP16 for training because it retains the same dynamic range as FP32. FP16 has limited dynamic range and often causes overflow in gradients. NVIDIA H100s and newer AMD GPUs natively support BF16 with no performance penalty.

Learning Rate Schedule: Warmup, Annealing, and Why

def get_cosine_schedule_with_warmup(
    optimizer: torch.optim.Optimizer,
    num_warmup_steps: int = 2000,
    num_training_steps: int = 100000,
) -> torch.optim.lr_scheduler.LRScheduler:

    def lr_lambda(current_step: int) -> float:
        if current_step < num_warmup_steps:
            # Linear warmup from 0 to 1
            return float(current_step) / max(1.0, num_warmup_steps)
        # Cosine decay from 1 to near-0
        progress = (current_step - num_warmup_steps) / max(
            1, num_training_steps - num_warmup_steps
        )
        return 0.5 * (1 + math.cos(math.pi * progress))

    return torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda)

Warmup exists because at initialization, the model's outputs are high-variance. A high learning rate in the first few steps can blow up the loss to infinity. Cosine annealing works because it reduces the learning rate gradually, allowing the model to settle into a deeper minimum rather than bouncing around the loss landscape.


Inference Optimization: Making It Fast Enough to Ship

Training is a one-time cost. Inference is perpetual. Every millisecond you shave off inference saves thousands of dollars in compute.

KV Caching: Why Your Model Has a "State"

During autoregressive generation, the model recomputes the Key and Value matrices for every previous token on every new token. KV caching stores these matrices so each token is computed exactly once.

class KVCache:
    """Stores Key and Value matrices from previous generation steps."""

    def __init__(self, max_batch_size: int, max_seq_len: int,
                 n_heads: int, d_k: int, dtype=torch.float16):
        self.cache_k = torch.zeros(
            max_batch_size, n_heads, max_seq_len, d_k, dtype=dtype, device="cuda"
        )
        self.cache_v = torch.zeros(
            max_batch_size, n_heads, max_seq_len, d_k, dtype=dtype, device="cuda"
        )
        self.seen_tokens = 0

    def update(self, k: torch.Tensor, v: torch.Tensor):
        """Append new K, V to cache."""
        batch_size = k.shape[0]
        new_tokens = k.shape[2]
        end = self.seen_tokens + new_tokens
        self.cache_k[:batch_size, :, self.seen_tokens:end] = k
        self.cache_v[:batch_size, :, self.seen_tokens:end] = v
        self.seen_tokens = end

    def get(self) -> tuple[torch.Tensor, torch.Tensor]:
        """Return cached K, V up to current position."""
        return (self.cache_k[:, :, :self.seen_tokens],
                self.cache_v[:, :, :self.seen_tokens])

Without KV caching, generating a 4096-token response would require 4096^2 / 2 = ~8.4 million attention computations — one for each pair of positions. With caching, it's 4096. That's a 2000x improvement.

The tradeoff: KV cache for a 70B model serving 4096-token contexts takes roughly 2GB of GPU memory per request. This is why context length and batch size compete for the same memory.

Speculative Decoding

Standard autoregressive generation generates one token at a time, bound by the model's forward pass. Speculative decoding uses a smaller, faster "draft" model to generate multiple tokens, then the big model verifies them in a single forward pass.

def speculative_decode(draft_model, target_model, prefix, n_speculate=5):
    """Draft tokens with small model, verify with large model."""

    # Step 1: Draft n tokens with small model
    draft_tokens = draft_model.generate(prefix, max_new_tokens=n_speculate)
    full_sequence = torch.cat([prefix, draft_tokens], dim=-1)

    # Step 2: Verify all at once with large model
    target_logits = target_model(full_sequence)
    accepted_tokens = [prefix]
    for i, (draft_id, logits) in enumerate(zip(draft_tokens[0], target_logits[0])):
        # Accept if the draft model's token has high probability in target
        draft_prob = F.softmax(logits, dim=-1)[draft_id]
        if draft_prob > 0.3:  # Threshold, tunable per model
            accepted_tokens.append(draft_id.item())
        else:
            # Sample from the target's distribution instead
            accepted_tokens.append(torch.multinomial(
                F.softmax(logits, dim=-1), 1
            ).item())
            break  # Can only accept up to the first rejection

    return torch.tensor([accepted_tokens])

A good draft model runs 5-10x faster than the target model. Typical acceptance rates are 60-80%, giving a 2-4x end-to-end speedup. The quality is identical to the target model since every draft is verified.

Quantization: Trading Precision for Speed

Production deployments almost never run models in FP32. The most common deployment format is INT4 or FP8, which reduces memory by 4-8x and increases throughput by 2-3x.

from transformers import AutoModelForCausalLM
import torch

# FP32: ~140GB for a 70B model (requires 2-4 GPUs)
model_fp32 = AutoModelForCausalLM.from_pretrained("model", torch_dtype=torch.float32)

# FP8: ~70GB for the same model (fits on 1-2 GPUs)
model_fp8 = AutoModelForCausalLM.from_pretrained(
    "model",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    quantization_config={"load_in_4bit": True},
)
# 4-bit: ~35GB (fits on 1 GPU)

The quality gap between FP16 and INT4 is surprisingly small. On benchmarks like MMLU, the difference averages 1-3 points. On real-world tasks, users rarely notice the difference. The throughput improvement from fitting on fewer GPUs is dramatic.


Production Serving Architecture

Single-GPU inference is for demos. Production serving needs to handle concurrent requests, batch intelligently, and survive hardware failures.

Continuous Batching

Traditional batching collects requests and processes them in fixed-size batches. Continuous batching dynamically adds and removes sequences from the running batch as they finish generation.

class ContinuousBatchScheduler:
    """Dynamically manages batch membership during generation."""

    def __init__(self, max_batch_size: int):
        self.active_sequences = []
        self.max_batch_size = max_batch_size

    def add_sequences(self, sequences: list[dict]):
        """Add new sequences if capacity exists."""
        available = self.max_batch_size - len(self.active_sequences)
        for seq in sequences[:available]:
            seq["kv_cache"] = KVCache(...)
            seq["generated_tokens"] = 0
            self.active_sequences.append(seq)

    def step(self):
        """Run one forward pass across all active sequences."""
        batch = self._pad_to_same_length(self.active_sequences)
        logits = model(batch)  # Single forward pass for all sequences

        finished = []
        for i, seq in enumerate(self.active_sequences):
            next_token = sample_token(logits[i, seq["generated_tokens"]])
            seq["tokens"].append(next_token)
            seq["generated_tokens"] += 1
            if self._should_stop(seq):
                finished.append(seq)

        # Remove finished sequences immediately
        for seq in finished:
            self.active_sequences.remove(seq)
            self._emit_response(seq)

        return self.active_sequences

Continuous batching increases throughput by 2-3x over fixed batching because it never waits for stragglers. A sequence that generates quickly leaves the batch early, freeing capacity for new requests.

Load Shedding

When request volume exceeds capacity, graceful degradation beats queue buildup.

class LoadShedder:
    """Drop requests when under pressure instead of building queues."""

    # Rate: 10 req/s → shed 0% | 50 req/s → shed 30% | 100 req/s → shed 70%
    def should_accept(self, request: dict) -> bool:
        current_qps = self._get_current_qps()
        target_qps = self._target_capacity()

        if current_qps <= target_qps:
            return True  # Under capacity, accept all

        # Gaussian probability shedding
        overflow = (current_qps - target_qps) / target_qps
        keep_prob = max(0.1, 1.0 - overflow)
        return random.random() < keep_prob

Every request that gets through under load gets full compute. Better to serve 60% of users well than 100% of users poorly.


The Memory Wall: Why Transformers Are Memory-Bound

Transformers are memory-bound, not compute-bound. The attention mechanism and FFN layers are dominated by matrix multiplications, and modern GPUs have more compute than memory bandwidth.

This means optimizing memory access often yields bigger gains than optimizing computation. Flash Attention, KV caching, and quantization all work primarily by reducing memory movement.

# Compute-bound vs memory-bound operations on an H100:
# Matrix multiply (4096 x 4096): ~3.5us — compute-limited
# Read 4096 tokens from HBM: ~12.5us — memory-limited
# Attention score computation: ~5us — balanced

The practical implication: for inference, moving a model to fewer GPUs (via quantization) almost always improves throughput per dollar, even though each GPU does the same amount of computation. The reduction in inter-GPU communication overhead dwarfs the per-GPU efficiency loss.


Image Suggestions

  1. Cover image (1400x788): A layered technical diagram showing a transformer block with attention heads radiating outward, code snippets on the left, and production metrics on the right. Orange/dark blue palette.

    • Alt text: "Transformer architecture diagram showing multi-head attention mechanism, feed-forward layers, and production deployment stack"
  2. Attention visualization: A heatmap showing how tokens attend to each other in a 12-layer transformer, with interpretable head patterns (subject-verb agreement, positional patterns, null heads).

    • Alt text: "Multi-head attention visualization showing learned attention patterns across 12 transformer layers"
  3. Training vs inference data flow diagram: Side-by-side comparison of the training pipeline (sharded datasets, mixed precision, gradient accumulation) and inference pipeline (KV cache, speculative decoding, continuous batching).

    • Alt text: "Comparison diagram of transformer training pipeline versus inference serving architecture with throughput metrics"
  4. Performance benchmark chart: Bar chart comparing inference throughput for FP32, FP16, FP8, and INT4 quantization formats on a 70B parameter model across batch sizes.

    • Alt text: "Benchmark chart comparing transformer inference throughput across FP32, FP16, FP8, and INT4 quantization formats"

My Take

Every transformer engineer I respect has the same arc: they start fascinated by the architecture, then spend years learning that the infrastructure around it matters more. The attention mechanism is elegant, but the training pipeline, the data loader, and the inference server are where the actual engineering happens.

If you're building with transformers today, the single best investment you can make is understanding your memory bandwidth utilization. Everything else — model architecture, training hyperparameters, deployment strategy — follows from knowing where your data sits and how fast it can move.


If you want the agent-level patterns that build on transformer inference, read AI Agent Implementation Mastery next.


SEO Verification:

  • SEO title: "Transformer Systems From Scratch" = 32 chars (limit: 60)
  • Subtitle: = 152 chars (limit: 160)
  • Meta description: A deep technical guide to building transformer-based systems from the ground up, covering attention mechanisms, training pipelines, optimization, and production serving infrastructure.
Back to all posts

Surface. Thanks for diving.

Shark model by alextkd2003, CC BY, converted to ASCII.