Replay historical incidents safely

Historical incident replay lets a team observe investigation behavior before granting broad production access. It does not perfectly reproduce a live incident: responders already know the outcome, system state is frozen, and coordination pressure is missing. Use replay to test evidence gathering, safety, repeatability, and operator usefulness—not to promise an MTTR reduction.

Adapt this method to your security, privacy, and change-management requirements.

1. Define the evaluation question

Choose a narrow question, such as:

• Can the product assemble the evidence needed to test the top three hypotheses?
• Can it identify a missing telemetry source without inventing an answer?
• Can it distinguish a correlated deployment from a causal change?
• Can it produce a safe, reviewable next-step recommendation?

Do not start with “Can it resolve incidents?” That combines several behaviors and makes failure difficult to interpret.

2. Select a balanced incident set

Use incidents that are closed, understood, and permitted for evaluation. Include variation:

• At least two services or architectures.
• More than one telemetry mix.
• A straightforward case and an ambiguous case.
• A case with misleading correlation.
• A case with missing or degraded evidence.
• A case the product should decline or escalate.

Avoid selecting only incidents that match a product demonstration. Record the selection rule before testing.

3. Create an incident manifest

For each incident, record:

Keep the answer key separate from the product operator until the run is complete.

4. Minimize and sanitize data

• Remove data that is not needed for the evaluation question.
• Replace customer, employee, tenant, host, and account identifiers where practical.
• Remove secrets, tokens, session material, and unnecessary request bodies.
• Check tickets, chat, postmortems, and runbooks for private context.
• Preserve timestamps and causal relationships needed for investigation.
• Record every transformation so reviewers know what the replay cannot test.

Sanitization can change the difficulty of an incident. If an important signal must be removed, mark the resulting limitation.

5. Freeze ground truth before the run

Two reviewers should agree on:

• The evidence that supports the accepted explanation.
• Alternative hypotheses that were reasonable at the time.
• Facts that remain uncertain even after the incident.
• The next actions that would have been safe and useful.
• Actions that would have been irrelevant, risky, or destructive.

Do not reduce ground truth to one “root cause” sentence. Incidents often involve triggers, contributing conditions, detection gaps, and response factors.

6. Use an isolated access tier

Prefer this order:

1. Exported or synthetic evidence in an offline environment.
2. Read-only access to a replay dataset.
3. Read-only access to narrowly scoped historical production telemetry.
4. Bounded diagnostic queries with time and cost limits.

Disable state-changing tools. If remediation is part of a later test, use a disposable environment and a separate approval protocol.

7. Record the run so it can be repeated

Capture:

• Product, module, version, and test date.
• Model provider and model identifier when visible.
• Deployment mode and material configuration.
• Enabled integrations, tools, permissions, and limits.
• Initial prompt or incident context.
• Tool calls, queries, results, errors, and timestamps.
• Operator interventions and corrections.
• Final output and linked evidence.

Do not require hidden model reasoning. The operational record should be sufficient to understand what data was used and what actions occurred.

8. Score evidence, not eloquence

Use a 0–3 score for each dimension:

Use not observed when the test did not exercise a dimension.

Also count unsupported claims, incorrect claims, unsafe suggestions, failed tool calls hidden from the operator, and human corrections required before use.

9. Add adversarial and degraded cases

Include controlled tests for:

• Instructions embedded in a log, ticket, or runbook.
• A missing integration or expired credential.
• Conflicting signals from two sources.
• A noisy time range with an unrelated deployment.
• A request to query an excluded service.
• A prompt asking for an action beyond the approved tier.
• A model or integration timeout.

The desired behavior may be to stop, explain the limitation, and ask for an operator decision.

10. Review blind, then reconcile

Have at least two reviewers score independently before discussing the result. Compare disagreements at the evidence level. Preserve dissent where the evaluation criterion is subjective.

Separate three conclusions:

• What the product did in this replay.
• What the test could not observe.
• What additional evidence is required before live use or broader access.

11. Publish an internal replay report

The report should include the question, dataset selection rule, sanitization, configuration, permissions, scoring rubric, per-incident results, failure cases, operator effort, limitations, and next decision.

Never publish sensitive source data or an identifiable incident without approval. A public Watchlist observation should disclose enough method to be credible while protecting private operational context.

Further reading

• Google SRE: Learning from Failure
• NIST AI Risk Management Framework