Data Quality Incident Write-Up Template

# Data quality incident: [short descriptive title]

## Status

- **State:** [investigating / mitigated / resolved / monitoring]
- **Severity:** [team definition]
- **Incident owner:** [name or team]
- **Started:** [YYYY-MM-DD HH:MM timezone]
- **Detected:** [YYYY-MM-DD HH:MM timezone]
- **Resolved:** [YYYY-MM-DD HH:MM timezone or open]
- **Last updated:** [YYYY-MM-DD HH:MM timezone]

## Executive summary

[In 3–5 sentences: what data was wrong or unavailable, who was affected, the decision/reporting risk, current status, and the most important next action.]

## Consumer impact

- **Affected consumers:** [teams, dashboards, reports, models, APIs]
- **Affected period:** [coverage]
- **Decision risk:** [what someone could have concluded or done]
- **Data exposure:** [incorrect / incomplete / delayed / duplicated / unavailable]
- **Confirmed usage:** [known views, exports, decisions, or “not yet confirmed”]

## Scope

### Affected

- [model.metric.column]

### Not affected

- [adjacent system checked and evidence]

### Unknown

- [uncertainty and next check]

## Detection

- **Signal:** [test, monitor, stakeholder report, reconciliation]
- **Expected behavior:** [contract or threshold]
- **Observed behavior:** [bounded evidence]
- **Why earlier controls did or did not catch it:** [specific gap]

## Timeline

| Time | Event | Evidence / decision |
|---|---|---|
| [time] | [source change, deployment, alert, report] | [link/query/log] |

## Immediate response

1. [Stopped, isolated, labeled, or rolled back the affected output.]
2. [Notified known consumers with scope and uncertainty.]
3. [Preserved evidence and identified the last known-good state.]

## Investigation

### Hypothesis 1: [statement]

- **Why plausible:** [evidence]
- **Smallest test:** [query/check]
- **Result:** [supported / rejected / unresolved]

### Hypothesis 2: [statement]

- **Why plausible:** [evidence]
- **Smallest test:** [query/check]
- **Result:** [supported / rejected / unresolved]

## Root cause

[Describe the technical and process conditions that produced the incident. Avoid naming a person as the cause.]

### Contributing factors

- [missing contract, ambiguous ownership, test gap, late source behavior, deployment condition]

## Resolution

- **Change made:** [code/config/data correction]
- **Records affected:** [bounded count and period]
- **Backfill or restatement:** [scope and method]
- **Consumer communication:** [what was sent, to whom, and when]

## Validation

- [ ] Reproduced the failure before the fix where safe.
- [ ] Confirmed the targeted test passes after the fix.
- [ ] Reconciled corrected totals to an independent source.
- [ ] Checked adjacent dates, entities, and downstream models.
- [ ] Confirmed dashboards/extracts refreshed from corrected data.
- [ ] Monitored for recurrence during [period].

## Prevention actions

| Action | Owner | Due date | Verification | Status |
|---|---|---|---|---|
| [specific control or contract] | [team] | [date] | [test/alert/review] | [open] |

## Communication record

- **Initial notice:** [time, audience, message link]
- **Scope update:** [time, audience, message link]
- **Resolution notice:** [time, audience, message link]

## Lessons

- **What worked:** [detection, response, collaboration]
- **What failed safely:** [control that limited impact]
- **What needs to change:** [system or process]
- **What remains uncertain:** [follow-up evidence]

## Appendix

- [queries]
- [logs]
- [screenshots with sensitive data removed]
- [related pull requests]
- [source contract or metric definition]

# Data quality incident: monthly payment value duplicated by refund join

## Status

- **State:** Resolved
- **Severity:** SEV-2 teaching example
- **Incident owner:** Analytics engineering
- **Started:** 2026-09-04 09:00 UTC
- **Detected:** 2026-09-04 13:20 UTC
- **Resolved:** 2026-09-04 16:10 UTC

## Executive summary

Monthly net payment value was overstated because successful payments were joined directly to refund events. Orders with multiple refunds repeated the payment amount. The affected model and dashboard extract were rebuilt after refunds were aggregated to order grain. No production business decision is represented by this fictional example.

## Consumer impact

- **Affected consumers:** Fictional finance dashboard
- **Affected period:** Months containing orders with multiple refunds
- **Decision risk:** Users could overstate collected value and misread month-over-month movement
- **Data exposure:** Duplicated amounts
- **Confirmed usage:** Dashboard views not verified in this example

## Detection

- **Signal:** Monthly total failed to reconcile to the independently aggregated payment and refund controls
- **Expected behavior:** Net payment value equals successful payments minus refunds
- **Observed behavior:** Payment amounts repeated once per linked refund event

## Root cause

The model joined order-grain payment totals to event-grain refunds and aggregated afterward. The query did not declare a target grain or test the measure before and after the join.

## Resolution

1. Aggregate refunds to one row per order.
2. Join order-grain payments and refunds to orders.
3. Rebuild affected months.
4. Reconcile the result to independent payment and refund totals.

## Validation

- [x] Reproduced the fan-out with an order containing two refunds.
- [x] Confirmed one row per order after the fix.
- [x] Reconciled monthly totals.
- [x] Checked orders with no refunds and multiple refunds.

## Prevention actions

| Action | Owner | Verification | Status |
|---|---|---|---|
| Add order-grain uniqueness test | Analytics engineering | CI test | Complete |
| Add monthly payment/refund reconciliation | Analytics engineering | Zero unexplained difference | Complete |
| Require grain in model documentation | Data team | Pull-request checklist | Open |

Related guidance.

• Data testing essentials→
• dbt learning path→
• Interview preparation→