What Is Inference? The Mental Model Every Backend Engineer Needs

As a backend engineer, you are accustomed to stateless request-response cycles or explicit database transactions. When you introduce Large Language Models (LLMs) into your stack, the abstraction often feels opaque. You send a prompt, and text appears. But under the hood, there is no "conversation." There is only a rigid, deterministic loop of linear algebra and probability.

This is Part 1 of the LLM Inference from Scratch series.

To build efficient, scalable AI systems, you must discard the anthropomorphic view of the model "thinking" and adopt the mechanical view of autoregressive generation. This guide breaks down the inference process into two distinct components: the Forward Pass (the computation) and the Autoregressive Loop (the iteration). Understanding this separation is essential for optimizing latency, managing memory, and debugging generation issues.

The Forward Pass: A Pure Function

At its core, a single step of LLM inference is a forward pass. In deep learning, this is the process of passing input data through the neural network’s layers to produce an output. For a Transformer-based model like GPT or Llama, this process is mathematically defined and stateless.

When you provide a sequence of tokens (e.g., [32, 102, 45, 9]) to the model, the forward pass executes the following mechanism:

1. Embedding: Each token ID is mapped to a dense vector representation (embedding) in a high-dimensional space. This vector captures semantic meaning based on the model's training.
2. Positional Encoding: Since transformers have no inherent sense of order, positional encodings are added to the embeddings to indicate each token's position in the sequence.
3. Transformer Layers: The embedded sequence passes through multiple layers. In each layer, two main operations occur:
   • Self-Attention: Each token attends to itself and all preceding tokens in the sequence, computing a weighted sum of values based on similarity. This allows the model to understand context (e.g., knowing that "it" in "The cat sat on the mat. It was comfortable" refers to the cat).
   • Feed-Forward Network (FFN): The output of the attention mechanism is passed through a non-linear transformation, allowing the model to learn complex patterns.
4. Logits Generation: The final output of the last layer is a matrix of raw scores, called logits, for every token in the model’s vocabulary.

Crucially, the forward pass is a pure function: $y = f(x)$. Given the same input sequence and the same model weights, it will always produce the same logits. There is no hidden state, no memory, and no "understanding" beyond the statistical patterns encoded in the weights.

Autoregressive Generation: The Loop

If the forward pass is a single step, inference is the repetition of that step. LLMs are autoregressive, meaning they generate text one token at a time, using the previous output as input for the next prediction. This is not a parallel generation of a full sentence; it is a sequential chain.

Consider the prompt: "Explain quantum physics."

1. Step 1: The model runs a forward pass on "Explain quantum physics.". It outputs a vector of logits for the entire vocabulary.
2. Sampling: The system applies a probability distribution (usually via softmax) to convert logits into probabilities. It then selects the next token. This selection can be deterministic (argmax) or stochastic (sampling with temperature). Let’s say it selects the token "Quantum".
3. Step 2: The model now constructs a new input sequence: "Explain quantum physics. Quantum". It runs the forward pass again.
4. Iteration: This process repeats until an end-of-sequence (EOS) token is generated or a maximum length is reached.

This loop is where the "magic" happens, but it’s also where latency accumulates. Each token requires a full forward pass. If you generate 100 tokens, you perform 100 forward passes. For backend engineers, this means inference latency is directly proportional to the output length.

The KV Cache: Optimization at the Hardware Level

A naive implementation of autoregressive generation would recompute the attention mechanism for the entire history at every step. For a sequence of length $N$, this results in $O(N^2)$ complexity per step, which is computationally prohibitive.

To optimize this, modern inference engines use the Key-Value (KV) Cache.

During the first forward pass, the model computes the Key and Value vectors for all tokens in the prompt. These vectors are stored in memory. In subsequent steps, when a new token is generated, the model only needs to compute the Key and Value vectors for that new token. It then retrieves the cached Keys and Values for all previous tokens and combines them.

This reduces the complexity of each subsequent step from $O(N^2)$ to $O(N)$. For backend engineers, this means:

• Memory Usage: The KV cache grows linearly with the sequence length. Long conversations consume significant GPU memory.
• Throughput: Caching allows for efficient batch processing. Multiple requests can share the same computational graph, with the KV cache handling the variable-length history for each.

Practical Implications for Backend Engineering

Understanding this mental model directly impacts how you design AI-powered services.

1. Latency Management: Since inference is a loop, you cannot parallelize the generation of individual tokens within a single request. However, you can parallelize batching. By grouping multiple user requests into a single batch, you utilize GPU parallelism to process the forward passes of different requests simultaneously.
2. Context Window Limits: The KV cache size is bounded by the model’s context window (e.g., 8K, 32K tokens). Exceeding this limit requires truncation or sliding window techniques. As a backend engineer, you must implement logic to manage these limits gracefully, perhaps by summarizing older parts of the conversation.
3. Cost Estimation: Compute cost is driven by the number of forward passes. Generating 100 tokens costs roughly 100x more than generating 1 token (excluding the initial prompt processing). Optimizing the prompt length and output length is critical for cost control.
4. Error Handling: Since the process is deterministic given the same seed and inputs, you can reproduce generation failures by inspecting the logits and sampling parameters. If the model generates nonsensical text, it’s often due to a high temperature setting or a poorly constructed prompt, not a "hallucination" in the mystical sense.

Conclusion

LLM inference is not a black box. It is a structured, iterative process of forward passes and autoregressive loops, optimized by mechanisms like the KV cache. For backend engineers, mastering this mental model allows you to reason about latency, memory, and cost with the same precision you apply to any other distributed system. The next time you integrate an LLM, remember: you’re not talking to a mind. You’re orchestrating a loop of matrix multiplications.