Prompt Assembly, Citations, and Grounding

This is Part 7 of the "RAG from Zero to Production" series.

In a Retrieval-Augmented Generation (RAG) system, the Large Language Model (LLM) is not merely a question-answering engine; it is a text completion engine operating under strict constraints. The reliability of the system depends entirely on how you perform prompt assembly—combining retrieved context with the user query. This assembly process must be mechanistically precise. If the context is poorly ordered or lacks explicit attribution, the model will hallucinate, filling gaps with its pre-training data rather than the provided facts. This post explores the mechanics of prompt assembly, context ordering, and grounding to build RAG applications that are accurate and auditable.

The Anatomy of a RAG Prompt

A RAG prompt is not a single string of text; it is a structured composition of distinct components. The most standard architecture separates these components into three distinct blocks: the System Instruction, the Context Block, and the User Query.

The System Instruction sets the behavioral constraints. It defines the role, the tone, and crucially, the grounding rules. For example, it explicitly states: "Answer only using the provided context. If the context does not contain the answer, state that you do not know." This prevents the model from relying on its internal weights for factual queries.

The Context Block contains the retrieved chunks. This is where the engineering challenge lies. You must inject raw text from your vector database or knowledge base into the prompt. However, simply concatenating chunks is insufficient. Each chunk must be identifiable so the model knows which part of the text supports which claim.

The User Query is the final input. It should be isolated from the context to ensure the model distinguishes between the source material and the question it needs to answer.

Consider this basic structure in code:

prompt_template = """
You are an expert assistant. Answer the question based only on the context below.
 
Context:
{context}
 
Question: {question}
 
Answer:
"""

While simple, this structure fails if {context} is a blob of unstructured text. The model cannot distinguish between Chunk A and Chunk B, making citation impossible.

Context Ordering and Recency Bias

LLMs exhibit a phenomenon often called the "lost in the middle" effect, where they pay disproportionate attention to information presented at the beginning and end of the context window, particularly in longer sequences. This is driven by the attention mechanism’s tendency to weight tokens at the start and end of the sequence more heavily than those in the middle, particularly as context length increases.

Therefore, the order in which you insert retrieved chunks into the prompt significantly impacts accuracy. There are two primary strategies:

1. Relevance-Based Ordering: Sort chunks by their similarity score from the vector search. The most relevant chunk goes first. This leverages the model’s tendency to anchor on the first piece of evidence. However, if the top result is slightly off-topic, the model may struggle to correct itself.
2. Chronological Ordering: Sort chunks by timestamp or document order. This is useful for narrative tasks where context builds over time.

For factual QA, relevance-based ordering is generally superior. But you must also manage the context window limit. If your retrieved context exceeds the model’s context window, you must truncate. Truncation should always happen from the end of the least relevant chunks, preserving the high-signal information at the front.

Grounding and Attribution Mechanisms

Grounding is the technical process of ensuring that every factual statement in the generated response can be traced back to a specific source in the context. Attribution is the user-facing manifestation of grounding: the citation.

Without explicit grounding, the model treats all text in the context block as a single monolithic truth. With grounding, you force the model to associate specific claims with specific source identifiers. This is achieved by embedding unique IDs into the context chunks.

For example, instead of passing raw text, you pass:

[Source: doc_123]
The capital of France is Paris.
[End Source]
 
[Source: doc_456]
Paris is located on the Seine River.
[End Source]

When the model generates the answer "Paris is the capital of France," the presence of [Source: doc_123] allows you to programmatically verify that the claim is grounded. If the model says "Paris is on the Amazon," you can detect a grounding failure because no source mentions the Amazon.

Attribution improves trust because it shifts the burden of verification from the user to the system. Users can click a citation to view the exact sentence in the source document that supports the claim. This is critical for enterprise applications where accuracy is non-negotiable.

Implementing Structured Citations

To implement citations effectively, you must constrain the model’s output format. You cannot rely on the model to "remember" to cite sources unless you explicitly instruct it to do so in the System Instruction and provide a clear output schema.

Here is a more advanced prompt assembly that enforces grounding:

system_prompt = """
You are an analytical assistant. 
1. Answer the question using ONLY the provided context.
2. Every factual claim must be followed by a citation in the format [Source: ID].
3. If the context does not contain the answer, reply with "NO_SOURCE_FOUND".
"""
 
user_prompt = """
Context:
{context}
 
Question: {question}
 
Answer:
"""

The {context} variable should be formatted as a list of strings, each prefixed with its ID. When parsing the LLM’s response, you can use regular expressions to extract citations and validate them against the available context IDs.

import re
 
def validate_citations(response, available_sources):
    citations = re.findall(r'\[Source: (\w+)\]', response)
    for cid in citations:
        if cid not in available_sources:
            raise ValueError(f"Invalid citation: {cid}")
    return True

This validation step is essential. It catches cases where the model hallucinates a source ID or misattributes information. By treating citations as data rather than decorative text, you create a feedback loop that ensures the system remains grounded.

Tradeoffs and Considerations

There are tradeoffs in implementing rigorous grounding. Adding source IDs and detailed instructions increases the token count of the prompt. This raises latency and cost. Furthermore, models have a limited capacity to track multiple sources simultaneously. If you provide 50 chunks with IDs, the model may confuse which ID corresponds to which fact.

A practical solution is to limit the number of chunks to the top 3–5 most relevant results. This balances the need for sufficient context with the model’s ability to maintain accurate attribution. Additionally, consider using models that have been fine-tuned for citation tasks, as they handle structured outputs more reliably than base models.

Conclusion

Prompt assembly in RAG is not just about feeding data to an LLM; it is about constructing a logical framework that constrains the model’s behavior. By separating system instructions, ordering context by relevance, and enforcing explicit grounding through source IDs, you transform a probabilistic text generator into a reliable information retrieval system. Citations are not just for show; they are the mechanism by which you verify accuracy and build user trust. As you move from prototype to production, prioritize the structural integrity of your prompts over the complexity of your retrieval algorithms. The best retrieval system in the world will fail if the prompt assembly does not clearly ground the output in the retrieved evidence. Proper attribution ensures that every claim can be traced back to its source, maintaining the integrity of the system.