Makhzon 360 · RB-LEDGERENGINE

Inventory Ledger Engine — incident runbook

Runbook hubMakhzon console

Inventory ledger writes are the source of truth for stock. Failed appends mean sales and receipts are not reflected in stock, so a data-integrity review is required after every incident here.

Warning at ledger write failure rate ≥ 1%, critical at ≥ 3%. Critical means user-visible impact: page the escalation owner.

dashboard: sim://grafana/ledger-enginelogs: sim://logs/ledger-engineowner: Inventory Teamrunbook: RB-LEDGERENGINE

Signals & thresholds

  • Primary metric: ledger write failure rate (%)
  • Warning threshold: ≥ 1%
  • Critical threshold: ≥ 3%
  • Dashboard: sim://grafana/ledger-engine
  • Log source: sim://logs/ledger-engine
  • SLA: 99.5% monthly availability
  • SLO: 99.99% successful ledger writes
  • External dependencies: Supabase PostgreSQL / APIs / Realtime

1. Detection

  1. Ledger write monitor reports ledger write failure rate above the warning threshold — writes are being rejected or rolled back.
  2. Downstream records appear missing or inconsistent to operators.

2. Triage

  1. Identify the failing operation type: sale-out, purchase-in, transfer, stocktake adjustment.
  2. Inspect the last failed ledger transactions and capture the exact database error.
  3. Check for constraint violations, deadlocks or serialization failures.
  4. Determine whether the data layer is also degraded — if so treat the database as the primary incident.

3. Mitigation

  1. Stop bulk import and stocktake jobs so retries do not compound the failure.
  2. If a specific store or item range is failing, isolate it and let the remaining traffic proceed.
  3. Re-drive failed ledger appends from the retry queue once the underlying error is cleared.

4. Verification

  1. Confirm ledger write failure rate is back under the warning threshold for at least 10 minutes.
  2. Reconcile stock on hand against ledger sums for affected items and locations.
  3. Confirm no orphaned sales or receipts exist without matching ledger entries.

5. Rollback / restore

  1. If a deploy caused the incident, roll back to the previous release and re-verify.
  2. If a configuration or threshold change caused it, restore the previous value and record why.
  3. Re-enable any job, feature flag or channel that was paused during mitigation, one at a time.

Escalation path

  • First responder: NOC on duty — acknowledge within 5 minutes.
  • Component owner: Inventory Team.
  • Escalation owner (page on critical): Inventory On-Call.
  • Escalate if the metric is still critical 15 minutes after mitigation, or if data integrity is in doubt.
  • Declare a major incident if two or more critical components are alerting at the same time.

Blast radius — business services affected (18)

Batch ManagementSerial Number ManagementInventory LedgerWarehouse InventoryStock TransferStocktake & AdjustmentLow-Stock AlertsSales / POSSales Returns / ReversalsPurchasingPurchase ReturnsB2B Transaction ConversionB2B ReceivingSupplier SalesRental ManagementInventory SynchronizationE-Commerce Order ImportPWA / Offline Operations

External dependencies

Supabase PostgreSQL / APIs / Realtime — Primary data plane, RPC and realtime channels

Simulated incidents that exercise this runbook

Supabase DB degradation — Primary Postgres data plane latency spike; RPC timeouts across engines.

Inventory ledger write failures — Ledger append transactions rolling back; stock movements not persisted.

Reporting notes

  • Record start time, actions taken and recovery time in the incident record.
  • Attach the dashboard screenshot and the dominant error message to the postmortem.
  • Content drafted for the monitoring demo — confirm the exact commands with the owning team before production use.