Business Context

ShieldNet operates endpoint and network monitoring for 8,000 enterprise customers and ingests millions of security events per day. The SOC wants an anomaly detection model that flags suspicious authentication and network activity early enough for analysts to investigate before a broader compromise occurs.

Dataset

You are given 90 days of event-level security telemetry aggregated into 5-minute entity windows (user-device or sourceIP-destinationIP pairs). The goal is to score each window as anomalous or normal.

• Size: 2.4M 5-minute windows, 42 features
• Target: Binary label for evaluation only — confirmed malicious activity (1) vs benign (0)
• Class balance: Extremely imbalanced — 0.35% malicious, 99.65% benign
• Missing data: 12% missing in endpoint telemetry for unmanaged devices; 4% missing in geo/IP enrichment

Success Criteria

A good solution should detect at least 70% of confirmed malicious windows while keeping precision high enough that analysts review no more than 300 alerts per day. The model should also provide enough signal to support triage and threshold tuning.

Constraints

• Near-real-time batch scoring every 5 minutes; p95 inference latency under 200 ms per 10K windows
• Limited labeled attack data, so the approach should work with mostly unlabeled data
• Analysts need interpretable anomaly reasons, not just a raw score

Deliverables

1. Propose an anomaly detection modeling approach and justify it.
2. Build a preprocessing and training pipeline for mixed numeric/categorical security features.
3. Define an evaluation strategy for highly imbalanced data and threshold selection.
4. Show how you would generate analyst-facing anomaly scores and top contributing features.
5. Describe how the model would be retrained and monitored in production.