How do you manage and optimize Adobe Analytics data pipelines moving into platforms like Databricks?
Using the provided Adobe Analytics event and Databricks batch tables, write a query that reports ingestion quality for successfully loaded batches. Treat the newest record for each logical event as authoritative and retain batches with no usable events.
batch_id, load_date, event_date, event_type, event_count, valid_event_count, total_revenue, and data_quality_pct.No events row for empty batches and Unmapped for unknown event codes. Sort by batch_id, event_date with nulls first, then event_type.| Column | Type | Description |
|---|---|---|
| raw_event_idPK | INT | Physical event record identifier |
| event_id | VARCHAR(30) | Logical Adobe event identifier |
| batch_id | VARCHAR(20) | Databricks load batch identifier |
| visitor_id | VARCHAR(30) | Adobe visitor identifier |
| event_timestamp | TIMESTAMP | Timestamp captured by Adobe Analytics |
| event_type_code | VARCHAR(20) | Adobe event type code |
| ingestion_timestamp | TIMESTAMP | Time the record entered the pipeline |
| is_valid | BOOLEAN | Whether validation passed |
| revenue | DECIMAL(12,2) | Revenue attributed to the event |
| Column | Type | Description |
|---|---|---|
| batch_idPK | VARCHAR(20) | Databricks pipeline batch identifier |
| load_date | DATE | Date the batch was loaded |
| load_status | VARCHAR(20) | Pipeline load status |
| target_table | VARCHAR(80) | Databricks destination table |
| Column | Type | Description |
|---|---|---|
| event_type_codePK | VARCHAR(20) | Adobe event type code |
| event_type_name | VARCHAR(40) | Readable event type name |