Segment Databricks Marketing Customers

Business Context

Databricks marketing wants a reliable customer segmentation framework for 420K accounts and leads across self-serve, commercial, and enterprise motions. The output will be used to personalize campaigns in Databricks Lakehouse Monitoring dashboards and improve conversion from marketing-qualified lead to pipeline.

Dataset

You are given an account-level dataset built in Delta Lake from 18 months of CRM, product usage, campaign engagement, and firmographic history.

• Rows: 420K accounts, 58 features
• Target for downstream validation: Binary — converted to qualified pipeline within 60 days
• Missing data: 18% missing in product usage for non-trial accounts, 11% missing in firmographics, 6% missing in engagement metrics
• Data quality issues: right-skewed spend and usage variables, high-cardinality categorical fields, and correlated activity metrics

Success Criteria

A strong solution should produce segments that are both analytically coherent and operationally useful:

• Stable clusters across monthly refreshes
• Clear business interpretation for campaign strategy
• Downstream lift over current rule-based segmentation on pipeline conversion
• A reproducible workflow in Databricks that can score new accounts weekly

Constraints

• Marketing stakeholders need interpretable segments, not black-box embeddings only
• Weekly batch scoring must finish in under 20 minutes on Databricks
• Segment definitions should be refreshable monthly without manual relabeling

Deliverables

1. Build an end-to-end segmentation pipeline, including preprocessing and clustering.
2. Select the number of segments using quantitative and business criteria.
3. Evaluate segment quality and stability, then validate usefulness against 60-day pipeline conversion.
4. Propose how to productionize scoring and monitoring in Databricks.