Context
NovaDocs, a SaaS collaboration platform, is launching a new in-app plugin marketplace. Product and analytics teams need a reliable event schema and pipeline to track plugin discovery, installation, activation, configuration, execution, and failure events across web, desktop, and backend services.
Today, interaction logs are emitted as inconsistent JSON payloads into application databases and daily ETL jobs load them into Snowflake. The current setup cannot support near-real-time adoption dashboards, experiment analysis, or reliable plugin-level usage metrics.
Scale Requirements
- Traffic: 120K events/sec peak, 25K events/sec average
- Sources: Web SDK, desktop client, plugin runtime service, billing API
- Event size: 1.5-3 KB JSON
- Latency target: < 3 minutes from emission to analytics-ready tables
- Retention: 180 days raw, 3 years aggregated
- Cardinality: 80M DAU, 250K plugins, 2B interaction events/day
Requirements
- Design an event schema that supports plugin lifecycle tracking:
view, install, enable, disable, configure, run, error, and uninstall.
- Model core entities and keys for
user, workspace, plugin, plugin_version, session, request_id, and event_id.
- Support schema evolution without breaking downstream consumers; include versioning and backward-compatible rules.
- Build a streaming + ELT pipeline that validates, deduplicates, enriches, and lands events into Snowflake.
- Create analytics-ready tables for plugin adoption funnels, daily active plugins, error rates, and workspace-level retention.
- Ensure idempotent reprocessing, backfills for late-arriving events, and replay from raw storage.
- Define monitoring for freshness, volume anomalies, schema drift, and data quality.
Constraints
- Infrastructure must stay on AWS using managed services where possible.
- Incremental budget is capped at $35K/month.
- GDPR/CCPA deletion requests must propagate across raw and curated layers within 72 hours.
- The team has strong SQL/dbt skills but limited experience operating custom Kafka clusters.
- Downstream BI consumers require stable table contracts and < 5% month-over-month schema churn.
