HMT Solutions 360 Monitoring — 58 technical service runbooks with escalation owner, SLA and SLO
Simulation environment. Runbook steps are drafted operational guidance for NOC training — confirm exact commands with each component owner before production use.
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Web / PWA & Mobile App Shell p95 page render latency · Synthetic browser check | RB-MWWEB | Frontend On-Call | 99.9% monthly availability | p95 render < 1500ms |
| Authentication & Verification Codes sign-in failure rate · Auth event monitor | RB-MWAUTH | Identity On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Turnstile Bot Protection challenge verification failure rate · Challenge verification monitor | RB-MWTURNSTILE | Security Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Roles & Privacy Rule Guards policy denial anomaly rate · Policy evaluation monitor | RB-MWAUTHZ | Security Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Supabase PostgreSQL Data Layer p95 query latency · Database health monitor | RB-MWDB | Data Platform On-Call | 99.95% monthly availability | p95 query < 350ms |
| Audit Log Service audit write failure rate · Audit write monitor | RB-MWAUDIT | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Zoho Email Delivery email failure rate · Email delivery monitor | RB-MWEMAIL | Notification On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Security Event Monitoring high-severity security events · Security event monitor | RB-MWSECURITY | Security On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Person Share Links & Requests share request failure rate · Share request monitor | RB-MWSHARING | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Viewer Pool & Egress Control daily viewer egress · Egress usage monitor | RB-MWVIEWERPOOL | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Abuse Reports & User Blocks open abuse reports · Moderation queue monitor | RB-MWABUSE | Trust & Safety | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Compliance Evidence & Exports evidence integrity failure rate · Evidence integrity monitor | RB-MWCOMPLIANCE | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Account Deletion Requests pending deletion requests · Deletion queue monitor | RB-MWDELETION | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Translation Service missing translation keys · Coverage monitor | RB-MWTRANSLATE | Localization Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Translation Runs translation run failure rate · Translation run monitor | RB-MWTRANSLATEAI | Localization Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| DNA Analysis Service DNA analysis failure rate · Analysis job monitor | RB-MWDNA | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Smart Hints Engine hint generation error rate · Hint generation monitor | RB-MWHINTS | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Historical Search Service p95 search latency · Search probe | RB-MWSEARCH | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Notification Engine notification delivery failure rate · Delivery monitor | RB-MWNOTIFY | Notification On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Media & Document Storage p95 object latency · Object storage probe | RB-MWSTORAGE | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Document Import Engine extraction failure rate · Extraction job monitor | RB-MWAIIMPORT | AI Platform On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Extraction Worker extraction queue depth · Worker queue monitor | RB-MWAIWORKER | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Import Review Service pending review requests · Review queue monitor | RB-MWAIREVIEW | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| AI Model Gateway Client model call failure rate · Model call monitor | RB-MWAIGATEWAY | AI Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Family Tree Engine p95 tree query latency · Tree RPC monitor | RB-MWTREEENGINE | Tree Platform On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Person & Relationship Service relationship write failure rate · Relationship write monitor | RB-MWPERSONENGINE | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tree Layout & Snapshot Service p95 layout build time · Layout build monitor | RB-MWTREELAYOUT | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tree Membership & Invitations tree invite failure rate · Invite delivery monitor | RB-MWTREEMEMBERSHIP | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tree Archive & Deletion Service pending tree deletions · Retention job monitor | RB-MWTREERETENTION | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Media Asset Service media upload failure rate · Upload pipeline monitor | RB-MWMEDIA | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Data Retention & Purge Jobs purge run failure rate · Purge run monitor | RB-MWPURGE | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Invoice & PDF Service invoice generation failure rate · Invoice generation monitor | RB-MWINVOICING | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| VAT / ZATCA Compliance Scan compliance check failure rate · Compliance scan monitor | RB-MWZATCA | Compliance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| VAT Return & Filing Service VAT filing mismatches · Filing monitor | RB-MWVAT | Compliance Engineering | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Image Proxy p95 image proxy latency · Proxy latency probe | RB-MWIMAGEPROXY | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Family Video Service video health check failure rate · Video health check | RB-MWVIDEO | Media Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Digest & Follow-up Jobs digest run failure rate · Job run monitor | RB-MWDIGEST | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Email Retry & Bounce Handling email retry queue depth · Retry queue monitor | RB-MWEMAILRETRY | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Push Notification Outbox push outbox backlog · Push fan-out monitor | RB-MWPUSH | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Announcements Dispatch announcement delivery failure rate · Dispatch monitor | RB-MWANNOUNCE | Notification Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Scheduled Job Runner scheduled job failure rate · Job run monitor | RB-MWJOBS | SRE On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Database Capacity database connections in use · Infrastructure snapshot | RB-MWDBCAPACITY | Data Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Incident Correlation Engine open incidents · Incident monitor | RB-MWINCIDENTS | SRE | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Infrastructure Snapshot Collector snapshot staleness · Snapshot freshness monitor | RB-MWMONITORING | SRE | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Admin Console Services admin action failure rate · Admin action monitor | RB-MWADMIN | Platform Governance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Exchange Rate Service exchange rate staleness · Rate refresh monitor | RB-MWFX | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Public Seerah Sharing public share error rate · Public share monitor | RB-MWSEERAH | Tree Platform | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Subscription & Entitlement Engine renewal failure rate · Renewal job monitor | RB-MWSUBSCRIPTION | Platform Finance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Moyasar Payment Client payment failure rate · Payment monitor | RB-MWMOYASAR | Platform Finance On-Call | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Apple / Google Store Billing receipt verification failure rate · Receipt verification monitor | RB-MWSTOREBILLING | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Refund & Dispute Service refund failure rate · Refund monitor | RB-MWREFUNDS | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Finance Reconciliation reconciliation mismatch rate · Reconciliation monitor | RB-MWRECON | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Webhook Intake & Quarantine quarantined webhook events · Quarantine monitor | RB-MWWEBHOOKS | Platform Finance | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Technical service | Runbook | Escalation owner | SLA | SLO |
|---|---|---|---|---|
| Support Desk SLA breach rate · SLA monitor | RB-MWSUPPORT | Support Operations | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Support Quality & Ratings open escalations · Quality monitor | RB-MWSUPPORTQUALITY | Support Operations | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Wedding Announcements & Reminders reminder dispatch failure rate · Reminder dispatch monitor | RB-MWWEDDINGS | Community Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Tribal Legacy Service legacy publish error rate · Legacy publish monitor | RB-MWTRIBAL | Community Team | 99.5% monthly availability | 99.9% of requests under warning threshold |
| Milestones & Reactions milestone event error rate · Event monitor | RB-MWMILESTONES | Community Team | 99.5% monthly availability | 99.9% of requests under warning threshold |