Evaluate Saved Search Feature Success

Business Context

Zillow has launched a Saved Search Alerts feature in its mobile app. Users can save a home search and receive notifications when new listings match their criteria. Four weeks after launch, product leadership wants to know whether the feature is successful and worth further investment.

Metric Scenario

In the first 28 days, 1.2M users viewed at least one search results page, 240K users tapped Save Search, and 168K completed alert setup. Of those who enabled alerts, 92K opened at least one alert notification, 41K returned to view a listing within 7 days of an alert, and 9.5K submitted a contact form to an agent. Overall app 4-week retention was 31% for users who never used the feature and 44% for users who did, but the PM notes that feature adopters may already be more engaged.

Stakeholders are asking which metrics should define success, which should be primary vs guardrail metrics, and how to separate feature adoption from true product impact.

Requirements

1. Define the core success metric for the feature and explain why it should be primary.
2. Propose a metric set covering adoption, engagement, retention, and downstream conversion.
3. Identify 3-5 guardrail metrics that should not worsen if the feature expands.
4. Describe how you would segment or decompose the metrics to diagnose weak performance.
5. Explain how you would distinguish correlation from causal impact when comparing users who adopted the feature vs those who did not.

Data Available

• search_events: search_id, user_id, query_filters, results_count, timestamp
• saved_search_events: user_id, search_id, save_clicked_at, alert_enabled_at, alert_frequency
• notification_events: user_id, notification_id, sent_at, opened_at, deep_link_destination
• listing_engagement: user_id, listing_id, view_at, favorite_at, share_at, contact_agent_at
• user_activity_daily: user_id, dt, sessions, searches, listings_viewed, retention_flag