The Reference Architecture for an LLM Application

Building a production-grade Large Language Model (LLM) application requires moving beyond simple prompt-and-response scripts. The architecture must handle non-determinism, high-latency inference, security constraints, and complex state management. A scalable reference architecture separates concerns into three distinct planes: the Control Plane (orchestration and logic), the Data Plane (retrieval and context enrichment), and the Gateway (routing and security). This separation allows each component to scale independently and fail gracefully.

The Control Plane: Orchestrating Determinism and Probability

The orchestrator is the central nervous system of the application. Its primary job is to manage the flow of execution between deterministic code (Python, Java, Go) and probabilistic model outputs. In any effective llm architecture, this separation is critical because unlike traditional APIs where if/else branches dictate flow, LLM applications often require the model itself to decide the next step, or the code to validate the model’s output before proceeding.

Mechanism: Stateful Chain Execution

The orchestrator maintains a conversation history and tool state. When a user request arrives, the orchestrator performs two critical steps:

1. Context Assembly: It retrieves relevant data from the Data Plane (see below).
2. Tool Definition: It exposes available functions (e.g., search_database, update_record) to the LLM via a structured schema (usually JSON Schema).

The LLM does not execute code directly. Instead, it returns a structured response indicating which tool to call and with what arguments. The orchestrator then validates these arguments against the schema, executes the tool in the secure backend environment, captures the result, and feeds it back into the LLM context for the final answer generation. This "ReAct" (Reasoning + Acting) pattern ensures that the LLM never hallucinates execution privileges.

// Example: Orchestrator receiving a tool call from LLM
{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_abc123",
      "type": "function",
      "function": {
        "name": "get_user_balance",
        "arguments": "{\"user_id\": \"usr_987\"}"
      }
    }
  ]
}

The orchestrator parses the arguments, calls the internal service, receives {"balance": 500}, and injects this into the next prompt turn. This mechanism isolates the LLM from direct database access, preventing injection attacks and ensuring transactional integrity.

The Data Plane: Retrieval-Augmented Generation (RAG)

The Data Plane handles the complex processing of context retrieval. In production, models are rarely prompted with raw training data; they are prompted with specific, retrieved chunks from enterprise knowledge bases. This layer must minimize latency while maximizing relevance.

Mechanism: Multi-Stage Retrieval Pipeline

A naive approach uses only vector similarity search. However, vector embeddings capture semantic meaning but not exact keyword matches or recency. A production Data Plane, a key component of the reference architecture, employs a multi-stage funnel:

1. Embedding Generation: The user query is converted into a dense vector using an embedding model (e.g., text-embedding-3-small).
2. Vector Search (Recall): A vector database (using HNSW or IVF indexes) retrieves the top K most similar documents. This is fast but noisy.
3. Cross-Encoder Reranking (Precision): The top K results are passed to a cross-encoder model (e.g., Cohere Rerank or BGE-Reranker). This model computes a relevance score for the query-document pair individually. It is computationally expensive, so it is only applied to the small subset of candidates, not the entire corpus.
4. Context Window Assembly: The top N reranked chunks are formatted into the prompt context.

This pipeline reduces hallucination by ensuring the LLM grounds its answers in verified, high-relevance snippets. The latency trade-off is acceptable because the reranker filters out irrelevant noise before the LLM pays the cost of processing it.

The Gateway Pattern: Routing, Safety, and Abstraction

Exposing an LLM endpoint directly to clients is a security and operational anti-pattern. A Gateway layer sits between the client and the orchestration layer, handling cross-cutting concerns. Within the reference architecture, this layer is essential for managing external traffic efficiently.

Mechanism: Classifier-Based Routing and Guardrails

The gateway performs three key functions:

1. Intent Routing: Not all queries require a heavy, expensive model. A lightweight classifier (or a small LLM) analyzes the intent. Simple factual queries might be routed to a fast, cheap model or even a traditional database lookup. Complex reasoning tasks are routed to a larger, more capable model. This optimizes cost and latency.
2. Input/Output Guardrails: Before the prompt reaches the orchestrator, the gateway scans for prompt injection attempts (e.g., "Ignore previous instructions"). After the LLM generates a response, the gateway scans for PII (Personally Identifiable Information) or toxic content. If detected, the response is blocked or sanitized.
3. Rate Limiting and Quotas: Since LLM tokens are metered, the gateway enforces per-user or per-application limits to prevent resource exhaustion.

POST /api/v1/gateway/chat
Content-Type: application/json
 
{
  "model": "auto", // Gateway decides which model to use
  "messages": [{"role": "user", "content": "What is the tax policy?"}],
  "metadata": {"tenant_id": "corp_a"}
}

The gateway returns a standardized response envelope, abstracting the underlying model complexity from the client. This allows the backend to swap models (e.g., from GPT-4 to Claude) without changing the client contract.

Observability: Tracing Non-Determinism

Traditional logging fails for LLMs because the same input can yield different outputs. To debug, you need distributed tracing that captures the full lifecycle of a request, including the prompt sent, the model used, the tokens consumed, and the latency at each stage. In a complete reference architecture, observability is not an afterthought but a foundational requirement.

Mechanism: Semantic Conventions

Adopt OpenTelemetry Semantic Conventions for LLMs. Each span should record:

• gen_ai.request.model: The specific model version.
• gen_ai.request.prompt: The full prompt text (sanitized).
• gen_ai.response.completion: The generated text.
• gen_ai.usage.total_tokens: For cost attribution.

By correlating these spans across the Gateway, Orchestrator, and Data Plane, you can identify bottlenecks. For example, if the Vector Search span takes 2 seconds, the bottleneck is the Data Plane, not the LLM inference.

Conclusion

A reference architecture for LLM applications is defined by its modularity. The orchestration layer manages state and tools, the Data Plane ensures factual grounding through retrieval, and the gateway enforces security and routing efficiency. By treating these as separate, testable components, architects can build systems that are not only intelligent but also reliable, secure, and cost-effective.