Mawrooth · RB-MWAIIMPORT

AI Document Import Engine — incident runbook

Runbook hubMawrooth console

The AI document import engine turns uploaded family documents into people and relationships. Failures here do not corrupt existing trees — every extraction waits for human review — but a sustained failure rate means members cannot import at all.

Warning at extraction failure rate ≥ 6%, critical at ≥ 15%. Critical means user-visible impact: page the escalation owner.

dashboard: sim://grafana/mw-ai-importlogs: sim://logs/mw-ai-importowner: AI Platformrunbook: RB-MWAIIMPORT

Signals & thresholds

  • Primary metric: extraction failure rate (%)
  • Warning threshold: ≥ 6%
  • Critical threshold: ≥ 15%
  • Dashboard: sim://grafana/mw-ai-import
  • Log source: sim://logs/mw-ai-import
  • SLA: 99.5% monthly availability
  • SLO: 99.9% of requests under warning threshold
  • External dependencies: Lovable AI Gateway, OpenAI API, Supabase Storage

1. Detection

  1. Extraction job monitor reports extraction failure rate above the warning threshold with errors returned by Lovable AI Gateway, OpenAI API, Supabase Storage.
  2. Provider status page or error codes confirm degraded service.

2. Triage

  1. Check whether failures come from the model gateway, the document itself, or the commit step.
  2. Confirm the uploaded file types and sizes in the failing jobs are within the supported limits.
  3. Compare failure counts per model to see whether one provider is degraded.
  4. Confirm the review queue is still draining — a stuck review queue looks like an import failure to members.

3. Mitigation

  1. Fail over to the secondary model provider if the primary gateway is degraded.
  2. Pause the extraction worker if partial extractions are being produced, so nothing incomplete reaches review.
  3. Re-queue affected jobs once the provider recovers; consent records and originals are retained.

4. Verification

  1. Run one known-good document end to end and confirm it reaches the review queue.
  2. Confirm no person was committed to a tree without an approved review record.

5. Rollback / restore

  1. If a deploy caused the incident, roll back to the previous release and re-verify.
  2. If a configuration or threshold change caused it, restore the previous value and record why.
  3. Re-enable any job, feature flag or channel that was paused during mitigation, one at a time.

Escalation path

  • First responder: NOC on duty — acknowledge within 5 minutes.
  • Component owner: AI Platform.
  • Escalation owner (page on critical): AI Platform On-Call.
  • Escalate if the metric is still critical 15 minutes after mitigation, or if data integrity is in doubt.
  • Declare a major incident if two or more critical components are alerting at the same time.

Blast radius — business services affected (1)

AI Document Import

External dependencies

Lovable AI Gateway — Document extraction, translation and hint generation models

OpenAI API — Secondary model provider for extraction and analysis

Supabase Storage — Document, media and attachment object storage

Simulated incidents that exercise this runbook

AI import failures — Document extraction jobs failing at the model gateway; imports queueing.

Reporting notes

  • Record start time, actions taken and recovery time in the incident record.
  • Attach the dashboard screenshot and the dominant error message to the postmortem.
  • Content drafted for the monitoring demo — confirm the exact commands with the owning team before production use.