Global Metrics Monitoring System

Design a large-scale metrics monitoring and alerting system for a global microservices architecture that lets engineers track service health, visualize performance via dashboards, and receive real-time alerts when anomalies occur.

Asked in the Stage 2 stage. Clarify scale, metric types, retention, alert latency, dashboard requirements, and notification channels before sizing the design.

Deliverables

1. Define functional and non-functional requirements and estimate ingestion, query, and alerting load.
2. Design metric collection, aggregation, storage, dashboard queries, and alert delivery.
3. Choose anomaly detection approaches for seasonal, sparse, and high-cardinality metrics.
4. Explain batch versus streaming processing, model training, evaluation, and deployment.
5. Address alert deduplication, incident correlation, missing data, feature drift, and training-serving skew.
6. Describe monitoring, capacity planning, failure handling, and rollback.

Constraints

Support global services, multi-tenant access control, configurable retention, low-latency alerts, cost-aware storage tiers, and graceful degradation during regional or pipeline failures.