Incident Response for AI Systems

In traditional Site Reliability Engineering (SRE), an incident is defined by a breach of Service Level Objectives (SLOs): the system is either up or down, fast or slow. In AI systems, particularly those powered by Large Language Models (LLMs), the binary nature of uptime collapses. An LLM can be fully available, responding in 200ms, yet produce a completely incorrect, biased, or harmful output. This is a functional outage disguised as health. For SREs and engineering managers, managing AI incidents requires a fundamental shift in detection, triage, and resolution mechanisms.

The Ontology of AI Incidents

To respond effectively, we must first classify what constitutes an incident in an AI context. Traditional outages are usually infrastructure-level (database connection pool exhaustion, network partition). AI incidents are often semantic or statistical.

The primary failure modes include:

1. Hallucination: The model generates plausible but factually incorrect information. This is a quality degradation, not a crash.
2. Prompt Injection: Adversarial inputs bypass safety filters or extract system instructions. This is a security breach specific to the generative interface.
3. Data Drift: The model’s performance degrades because the input distribution has changed since training (e.g., new slang, new regulatory requirements).
4. Output Poisoning: In multi-agent systems, one agent’s faulty output becomes the input for another, causing cascading errors.

Standard monitoring tools (Datadog, New Relic) track CPU, memory, and latency. They do not track semantic correctness. Therefore, the first step in AI IR is establishing observability for quality, not just availability. Modern evaluation frameworks leverage techniques such as LLM-as-a-judge [1] and embedding similarity metrics [2] to quantify these semantic failures. Furthermore, architectures employing Retrieval-Augmented Generation (RAG) [3] introduce additional failure points where the retrieval layer may fail to provide relevant context, exacerbating hallucination rates.

Detection: The Golden Dataset Alert

The mechanism for detecting AI incidents is continuous evaluation against a "Golden Dataset." This is a static, high-quality set of input-output pairs that represent ground truth. Unlike unit tests that check for exact string matches, AI evaluation uses LLM-as-a-judge or embedding similarity to score responses.

When a new model version or prompt is deployed, it is run against this dataset. If the score drops below a threshold (e.g., 95% faithfulness), an alert fires. This is the "canary" phase of AI deployment.

Consider the following alerting mechanism:

# Pseudocode for Quality Alerting Trigger
def on_model_deployment(model_version, prompt_template):
    results = evaluate_against_golden_dataset(model_version, prompt_template)
    if results.accuracy_score < SLA_THRESHOLD:
        trigger_alert(
            severity="critical",
            message=f"Quality degradation detected: {results.accuracy_score}",
            artifact=model_version,
            action="rollback"
        )

This alert is not based on system load. It is based on semantic fidelity. If the alert triggers, the incident is classified as a Quality Degradation Event. The immediate response is not to restart servers, but to freeze the deployment pipeline and roll back to the previous model version or prompt template.

Incident Response: Detection via Golden Datasets

Effective incident response begins with robust detection. While the previous section outlined the technical mechanism, the operational discipline requires integrating these checks into a broader LLMOps workflow. By treating evaluation metrics as first-class citizens in the CI/CD pipeline, teams can automate the initial triage of potential incidents. This proactive stance ensures that quality degradation is caught before it reaches production users, forming the backbone of a mature incident response strategy for generative AI.

The AI Runbook: Triage and Mitigation

A runbook is a step-by-step procedure for handling specific incidents. For AI, the runbook must account for the non-deterministic nature of models. Here is a concrete scenario: A customer support chatbot begins giving incorrect refund policies.

Step 1: Triage

The SRE receives the alert. The first question is: Is this a security issue or a quality issue?

• Security Check: Run the last 100 failed prompts through a toxicity/injection detector. If injection is detected, activate the WAF (Web Application Firewall) rules for LLMs.
• Quality Check: If no injection, compare the current prompt template against the previous version. Did a developer change the system instructions?

Step 2: Mitigation

Mitigation strategies for AI are different from code fixes. You cannot "patch" a neural network in production. You must mitigate via configuration.

1. Rollback Prompt Version: If the incident started after a prompt update, revert to the previous prompt version. This is the fastest fix.
2. Activate Guardrails: If the model is hallucinating facts, enable a retrieval-augmented generation (RAG) layer that forces the model to cite sources. If it cannot cite, it must say "I don't know."
3. Temperature Reduction: Lower the temperature parameter (e.g., from 0.7 to 0.2). This reduces creativity and increases determinism, which may reduce hallucinations at the cost of flexibility.

# Example: Incident Mitigation Configuration
incident_type: hallucination
mitigation_steps:
  - action: rollback_prompt
    target: "prompt_v2.1"
  - action: adjust_parameters
    key: temperature
    value: 0.1
  - action: enable_guardrails
    module: "fact_check_rag"
    strict_mode: true

Step 3: Verification

After mitigation, verify the fix using the same Golden Dataset. Do not rely on user feedback alone; it is too slow and noisy. Automated evaluation must confirm that the accuracy score has returned to baseline.

Postmortems and Model Decay

A postmortem for an AI incident should not focus on "who wrote the bad code." The model weights are opaque. Instead, the postmortem must focus on data lineage and prompt versioning.

Key Questions for AI Postmortems

1. Data Provenance: Did the training data contain biases or errors that were amplified by the prompt?
2. Prompt Drift: Was the system prompt modified without adequate testing against the Golden Dataset?
3. Evaluation Coverage: Did the Golden Dataset include edge cases that triggered the failure? If not, the evaluation suite is incomplete.

The Concept of Model Decay

Unlike traditional software, AI models degrade over time. This is known as model decay. The world changes; the model’s static knowledge base becomes outdated. This is a recurring incident type. The postmortem should recommend implementing a scheduled retraining or fine-tuning pipeline based on recent user interactions.

For example, if the chatbot fails to answer questions about a new product launched last week, the postmortem should mandate an update to the vector database (RAG index) and a re-evaluation of the model’s ability to retrieve that specific information.

Conclusion

Incident Response for AI systems is not an extension of traditional SRE; it is a new discipline. The core difference is the shift from monitoring system health to monitoring semantic correctness. By implementing automated evaluation against Golden Datasets, creating runbooks that prioritize prompt rollback and guardrail activation, and conducting postmortems focused on data and prompt lineage, SREs can manage the inherent uncertainty of AI systems. The goal is not to eliminate errors, but to detect quality degradation before it impacts the user experience.

This approach transforms AI from a black box into a manageable service, where incidents are predictable, detectable, and resolvable through structured operational practices. Effective incident response in this domain requires continuous adaptation, ensuring that our operational frameworks evolve alongside the capabilities of the models we deploy.

[1] Zheng, L., et al. "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena." arXiv preprint arXiv:2306.05685 (2023). [2] Reimers, N., & Gurevych, I. "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks." Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (2019). [3] Lewis, P., et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks." Advances in Neural Information Processing Systems (2020).