Rare Event Detection Under Imbalance

Scenario

You're evaluating a classifier for rare system anomalies or rider complaints, where positives are much less common than negatives. You need to decide how to assess model quality and how to handle the imbalance without misleading yourself.

Question

How do you handle highly imbalanced datasets when trying to predict rare system anomalies or rider complaints?