HMT Solutions 360 Monitoring — 108 technical service runbooks with escalation owner, SLA and SLO
Simulation environment. Runbook steps are drafted operational guidance for NOC training — confirm exact commands with each component owner before production use.
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Supplier Governance Engine verification queue depth · Governance review monitor | RB-SUPPLIERGOV | Marketplace Ops | 99.5% monthly availability | 99.9% of requests under warning threshold |
| B2B Marketplace Engine marketplace RPC failure rate · Marketplace RPC monitor | RB-B2BENGINE | Marketplace On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Marketplace Commission Engine commission calc failure rate · Commission calc monitor | RB-COMMISSIONENGINE | Marketplace Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Platform Finance Engine reconciliation mismatch rate · Reconciliation monitor | RB-PLATFORMFINANCE | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| ZATCA Fatoora API Client ZATCA API failure rate · Clearance probe | RB-ZATCACLIENT | Compliance On-Call | 99% clearance within 5 minutes | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| CRM Engine campaign processing error rate · Campaign processor monitor | RB-CRMENGINE | CRM Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Loyalty Engine loyalty posting failure rate · Points ledger monitor | RB-LOYALTYENGINE | CRM Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Customer Account Engine p95 customer query latency · Customer RPC monitor | RB-CUSTOMERENGINE | CRM Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Credit & Aging Engine aging recalculation error rate · Aging recalculation monitor | RB-CREDITENGINE | Finance Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Collections Engine follow-up queue depth · Follow-up queue monitor | RB-COLLECTIONSENGINE | Collections Ops | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| PWA Offline / Sync Service offline sync queue depth · Sync queue monitor | RB-PWASYNC | Frontend Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| E-Commerce Adapter Layer stock sync lag · Sync worker monitor | RB-ECOMADAPTER | Integrations On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Public API / API Key Service API 5xx rate · Edge API monitor | RB-PUBLICAPI | Integrations Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Webhook Dispatcher webhook delivery failure rate · Delivery queue monitor | RB-WEBHOOKDISPATCH | Integrations Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Error & Health Monitoring reported errors (24h) · Error intake monitor | RB-ERRORMONITOR | SRE On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Accounting Posting Guard posting errors · Posting error monitor | RB-ACCOUNTINGGUARD | Finance Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Product Catalog Engine p95 catalog query latency · Catalog RPC monitor | RB-CATALOGENGINE | Inventory Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Inventory Ledger Engine ledger write failure rate · Ledger write monitor | RB-LEDGERENGINE | Inventory On-Call | 99.5% monthly availability | 99.99% successful ledger writes |
| Barcode Service scan decode failure rate · Scan pipeline monitor | RB-BARCODESVC | Inventory Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Email Service email failure rate · Delivery monitor | RB-EMAILSVC | Notification On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Push Notification Service push failure rate · Push fan-out monitor | RB-PUSHSVC | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Performance / RPC Monitoring p95 RPC latency · RPC metrics collector | RB-PERFMONITOR | SRE | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Search Service p95 search latency · Search probe | RB-SEARCHSVC | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Platform Governance / Audit platform audit write failure rate · Platform audit monitor | RB-PLATFORMGOV | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Data Import Service import batch failure rate · Import batch monitor | RB-IMPORTSVC | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Store Alert Service open store alerts · Alert intake monitor | RB-ALERTSVC | Store Operations | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Web / PWA Presentation p95 page render latency · Synthetic browser check | RB-WEBPWA | Frontend On-Call | 99.9% monthly availability | p95 render < 1500ms for 99% of sessions |
| Authentication Client login failure rate · Auth flow probe | RB-AUTHCLIENT | Identity On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tenant / Active Store Context context resolution error rate · Context resolution probe | RB-TENANTCTX | Platform Core | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Store / Branch Service p95 RPC latency · RPC monitor | RB-STOREBRANCH | Platform Core | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Supabase PostgreSQL Data Layer p95 query latency · Database health monitor | RB-DBLAYER | Data Platform On-Call | 99.95% monthly availability | p95 query < 350ms |
| Authorization / Permission Guards policy denial anomaly rate · Policy evaluation monitor | RB-AUTHZ | Security Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Audit Service audit write failure rate · Audit write monitor | RB-AUDITSVC | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Warehouse / Location Engine p95 transfer RPC latency · Transfer RPC monitor | RB-WAREHOUSEENGINE | Inventory Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Reporting & Analytics Service p95 report build time · Report job monitor | RB-REPORTINGSVC | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Platform Administration Service admin action failure rate · Admin action monitor | RB-PLATFORMADMIN | Platform Governance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Supplier Master Service p95 supplier query latency · Supplier RPC monitor | RB-SUPPLIERMASTER | Procurement Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Procurement Engine PO transition failure rate · PO lifecycle monitor | RB-PROCUREMENTENGINE | Procurement Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Supplier Account / Payment Engine supplier posting failure rate · Supplier ledger monitor | RB-SUPPLIERACCOUNT | Finance Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Rental Engine rental cycle error rate · Rental cycle monitor | RB-RENTALENGINE | Sales Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Rental Contract Signature Service signature OTP failure rate · Signature OTP monitor | RB-RENTALSIGNATURE | Sales Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Subscription Engine renewal failure rate · Renewal job monitor | RB-SUBSCRIPTIONENGINE | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Moyasar Payment Gateway Client payment gateway failure rate · Charge probe | RB-MOYASARCLIENT | Platform Finance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Sales / POS Engine checkout failure rate · Checkout transaction monitor | RB-POSENGINE | Sales On-Call | 99.9% monthly availability | 99.9% of requests under warning threshold |
| Payment Recording / Allocation allocation error rate · Allocation job monitor | RB-PAYMENTALLOC | Finance Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Promotions Engine p95 rule evaluation latency · Rule evaluation monitor | RB-PROMOENGINE | Sales Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Accounting / GL Engine journal posting error rate · Journal posting monitor | RB-GLENGINE | Finance On-Call | 99.5% monthly availability | 99.99% balanced journal postings |
| VAT / ZATCA Document Engine document generation error rate · UBL generation monitor | RB-ZATCADOC | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Support / SLA Service SLA breach rate · SLA breach monitor | RB-SUPPORTSVC | Support Operations | 99.5% monthly availability | 99.9% of requests under warning threshold |
| SMS Service SMS outbox backlog · SMS outbox monitor | RB-SMSSVC | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Web / PWA & Mobile App Shell p95 page render latency · Synthetic browser check | RB-MWWEB | Frontend On-Call | 99.9% monthly availability | p95 render < 1500ms |
| Authentication & Verification Codes sign-in failure rate · Auth event monitor | RB-MWAUTH | Identity On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Turnstile Bot Protection challenge verification failure rate · Challenge verification monitor | RB-MWTURNSTILE | Security Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Roles & Privacy Rule Guards policy denial anomaly rate · Policy evaluation monitor | RB-MWAUTHZ | Security Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Supabase PostgreSQL Data Layer p95 query latency · Database health monitor | RB-MWDB | Data Platform On-Call | 99.95% monthly availability | p95 query < 350ms |
| Audit Log Service audit write failure rate · Audit write monitor | RB-MWAUDIT | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Zoho Email Delivery email failure rate · Email delivery monitor | RB-MWEMAIL | Notification On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Security Event Monitoring high-severity security events · Security event monitor | RB-MWSECURITY | Security On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Person Share Links & Requests share request failure rate · Share request monitor | RB-MWSHARING | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Viewer Pool & Egress Control daily viewer egress · Egress usage monitor | RB-MWVIEWERPOOL | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Abuse Reports & User Blocks open abuse reports · Moderation queue monitor | RB-MWABUSE | Trust & Safety | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Compliance Evidence & Exports evidence integrity failure rate · Evidence integrity monitor | RB-MWCOMPLIANCE | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Account Deletion Requests pending deletion requests · Deletion queue monitor | RB-MWDELETION | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Translation Service missing translation keys · Coverage monitor | RB-MWTRANSLATE | Localization Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Translation Runs translation run failure rate · Translation run monitor | RB-MWTRANSLATEAI | Localization Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| DNA Analysis Service DNA analysis failure rate · Analysis job monitor | RB-MWDNA | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Smart Hints Engine hint generation error rate · Hint generation monitor | RB-MWHINTS | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Historical Search Service p95 search latency · Search probe | RB-MWSEARCH | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Notification Engine notification delivery failure rate · Delivery monitor | RB-MWNOTIFY | Notification On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Media & Document Storage p95 object latency · Object storage probe | RB-MWSTORAGE | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Document Import Engine extraction failure rate · Extraction job monitor | RB-MWAIIMPORT | AI Platform On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Extraction Worker extraction queue depth · Worker queue monitor | RB-MWAIWORKER | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Import Review Service pending review requests · Review queue monitor | RB-MWAIREVIEW | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Model Gateway Client model call failure rate · Model call monitor | RB-MWAIGATEWAY | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Family Tree Engine p95 tree query latency · Tree RPC monitor | RB-MWTREEENGINE | Tree Platform On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Person & Relationship Service relationship write failure rate · Relationship write monitor | RB-MWPERSONENGINE | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tree Layout & Snapshot Service p95 layout build time · Layout build monitor | RB-MWTREELAYOUT | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tree Membership & Invitations tree invite failure rate · Invite delivery monitor | RB-MWTREEMEMBERSHIP | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tree Archive & Deletion Service pending tree deletions · Retention job monitor | RB-MWTREERETENTION | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Media Asset Service media upload failure rate · Upload pipeline monitor | RB-MWMEDIA | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Data Retention & Purge Jobs purge run failure rate · Purge run monitor | RB-MWPURGE | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Invoice & PDF Service invoice generation failure rate · Invoice generation monitor | RB-MWINVOICING | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| VAT / ZATCA Compliance Scan compliance check failure rate · Compliance scan monitor | RB-MWZATCA | Compliance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| VAT Return & Filing Service VAT filing mismatches · Filing monitor | RB-MWVAT | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Image Proxy p95 image proxy latency · Proxy latency probe | RB-MWIMAGEPROXY | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Family Video Service video health check failure rate · Video health check | RB-MWVIDEO | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Digest & Follow-up Jobs digest run failure rate · Job run monitor | RB-MWDIGEST | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Email Retry & Bounce Handling email retry queue depth · Retry queue monitor | RB-MWEMAILRETRY | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Push Notification Outbox push outbox backlog · Push fan-out monitor | RB-MWPUSH | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Announcements Dispatch announcement delivery failure rate · Dispatch monitor | RB-MWANNOUNCE | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Scheduled Job Runner scheduled job failure rate · Job run monitor | RB-MWJOBS | SRE On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Database Capacity database connections in use · Infrastructure snapshot | RB-MWDBCAPACITY | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Incident Correlation Engine open incidents · Incident monitor | RB-MWINCIDENTS | SRE | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Infrastructure Snapshot Collector snapshot staleness · Snapshot freshness monitor | RB-MWMONITORING | SRE | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Admin Console Services admin action failure rate · Admin action monitor | RB-MWADMIN | Platform Governance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Exchange Rate Service exchange rate staleness · Rate refresh monitor | RB-MWFX | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Public Seerah Sharing public share error rate · Public share monitor | RB-MWSEERAH | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Subscription & Entitlement Engine renewal failure rate · Renewal job monitor | RB-MWSUBSCRIPTION | Platform Finance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Moyasar Payment Client payment failure rate · Payment monitor | RB-MWMOYASAR | Platform Finance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Apple / Google Store Billing receipt verification failure rate · Receipt verification monitor | RB-MWSTOREBILLING | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Refund & Dispute Service refund failure rate · Refund monitor | RB-MWREFUNDS | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Finance Reconciliation reconciliation mismatch rate · Reconciliation monitor | RB-MWRECON | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Webhook Intake & Quarantine quarantined webhook events · Quarantine monitor | RB-MWWEBHOOKS | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Support Desk SLA breach rate · SLA monitor | RB-MWSUPPORT | Support Operations | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Support Quality & Ratings open escalations · Quality monitor | RB-MWSUPPORTQUALITY | Support Operations | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Wedding Announcements & Reminders reminder dispatch failure rate · Reminder dispatch monitor | RB-MWWEDDINGS | Community Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tribal Legacy Service legacy publish error rate · Legacy publish monitor | RB-MWTRIBAL | Community Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Milestones & Reactions milestone event error rate · Event monitor | RB-MWMILESTONES | Community Team | 99.5% monthly availability | 99.9% of requests under warning threshold |