How would you solve a data manipulation problem where the data is corrupted and half the columns have a poor fill rate?
Using the provided raw_claims table, produce a column-level data quality report. Treat blank strings as missing, retain the latest row for duplicate nonblank claim IDs, and identify invalid values using the column rules represented by the sample data.
column_name, row_count, nonblank_count, fill_rate_pct, invalid_count, poor_fill_rate, and recommendation.| Column | Type | Description |
|---|---|---|
| raw_row_idPK | INT | Unique identifier for the raw ingested row |
| claim_id | VARCHAR(30) | Claim identifier, potentially duplicated or blank |
| member_id | VARCHAR(30) | Member identifier |
| service_date | VARCHAR(20) | Service date stored as raw text |
| claim_status | VARCHAR(20) | Raw claim status |
| billed_amount | VARCHAR(30) | Billed amount stored as raw text |
| diagnosis_code | VARCHAR(20) | Diagnosis code expected to match the sample format |
| ingested_at | TIMESTAMP | Timestamp when the raw row was ingested |