Production Cluster Ingress Recovery Runbook
Summary: Standard operating procedure for edge proxy failover and BGP route redirection during traffic spikes.
Document Notice
Maintained by Elena Rostova (Lead Site Reliability Engineer) for the DevOps & SRE Knowledge Base team. Version: v2.4.0.
1. Architecture Overview
When external ingress proxies experience latency degradation exceeding 120ms, automatic failover diverts traffic to the secondary edge cluster in DevOps & SRE Knowledge Base.
# Verify current ingress connection health
kubectl get ingress -n production -o wide
curl -I -s -o /dev/null -w "%{http_code}" https://ingress.edge.internal/healthz
2. Automated Escalation Matrix
| Severity | Trigger Condition | On-Call Contact |
|---|---|---|
| SEV-1 | Packet loss > 5% across edge nodes | elena.rostova@devwiki.internal |
| SEV-2 | p99 Latency > 250ms for 3 minutes | Secondary SRE Pager |
3. Verification & Metrics Verification
Ensure that connection pools have returned to nominal capacity before resolving the PagerDuty incident record.