Mawrooth · RB-MWWEBHOOKS

Webhook Intake & Quarantine — incident runbook

Runbook hubMawrooth console

Payment and store notifications arrive as webhooks. Anything unsigned or unprocessable is quarantined rather than applied, so a growing quarantine means entitlements may be lagging behind real purchases.

Warning at quarantined webhook events ≥ 5 items, critical at ≥ 15 items. Critical means user-visible impact: page the escalation owner.

dashboard: sim://grafana/mw-webhookslogs: sim://logs/mw-webhooksowner: Platform Financerunbook: RB-MWWEBHOOKS

Signals & thresholds

  • Primary metric: quarantined webhook events (items)
  • Warning threshold: ≥ 5 items
  • Critical threshold: ≥ 15 items
  • Dashboard: sim://grafana/mw-webhooks
  • Log source: sim://logs/mw-webhooks
  • SLA: 99.5% monthly availability
  • SLO: 99.9% of requests under warning threshold
  • External dependencies: Moyasar Payment Gateway, Apple App Store Billing, Google Play Billing

1. Detection

  1. Quarantine monitor reports quarantined webhook events above the warning threshold.
  2. Error volume for this component rises in the log source and error reports.

2. Triage

  1. List quarantined events by provider and reason to separate signature failures from processing failures.
  2. Check whether a provider changed its signing secret or payload shape.
  3. Cross-check affected users against their subscription state to size the impact.

3. Mitigation

  1. Fix the signing secret or handler defect first — never replay into a broken handler.
  2. Replay quarantined events once, in order, and confirm each resolves.
  3. Reconcile subscriptions and entitlements for the affected users afterwards.

4. Verification

  1. Confirm the quarantine count returns to zero and no event is replayed twice.
  2. Confirm entitlement mismatches for affected users are cleared.

5. Rollback / restore

  1. If a deploy caused the incident, roll back to the previous release and re-verify.
  2. If a configuration or threshold change caused it, restore the previous value and record why.
  3. Re-enable any job, feature flag or channel that was paused during mitigation, one at a time.

Escalation path

  • First responder: NOC on duty — acknowledge within 5 minutes.
  • Component owner: Platform Finance.
  • Escalation owner (page on critical): Platform Finance.
  • Escalate if the metric is still critical 15 minutes after mitigation, or if data integrity is in doubt.
  • Declare a major incident if two or more critical components are alerting at the same time.

Blast radius — business services affected (3)

Card PaymentsApp Store PurchasesWebhook Intake

External dependencies

Moyasar Payment Gateway — Subscription charges, refunds and callbacks

Apple App Store Billing — In-app purchase receipts and server notifications

Google Play Billing — In-app purchase receipts and developer notifications

Simulated incidents that exercise this runbook

Payment gateway failures — Charge creation and webhook verification failing; renewals not completing.

App store billing failures — Apple and Google receipt verification failing; store entitlements out of sync.

Reporting notes

  • Record start time, actions taken and recovery time in the incident record.
  • Attach the dashboard screenshot and the dominant error message to the postmortem.
  • Content drafted for the monitoring demo — confirm the exact commands with the owning team before production use.