PySpark Data Cleaning and EDA | Dataford Interview Questions

Tell me about how you would clean a dataset and perform descriptive data analysis using PySpark.

For this SQL exercise, write a PostgreSQL query that cleans the transaction records and summarizes valid completed transactions by customer segment. Treat duplicate transaction IDs as one record, keeping the latest ingestion, and exclude invalid amounts or dates.

Output

One row per customer segment, including Unknown when no usable segment exists
Columns: customer_segment, transaction_count, total_amount, average_amount, minimum_amount, maximum_amount
Include only valid completed transactions with positive amounts and valid dates
Order by total_amount descending, then customer_segment ascending

Schema

raw_transactions

Column	Type	Description
row_idPK	INT	Unique physical row identifier
transaction_id	VARCHAR(20)	Logical transaction identifier, which can be duplicated across ingestions
customer_id	VARCHAR(20)	Customer identifier from the source transaction data
amount_text	VARCHAR(40)	Transaction amount stored as raw text
status_text	VARCHAR(30)	Raw transaction status
transaction_date_text	VARCHAR(20)	Transaction date stored as raw text
ingested_at	TIMESTAMP	Timestamp when the source record was ingested

customer_reference

Column	Type	Description
customer_idPK	VARCHAR(20)	Customer identifier
segment	VARCHAR(40)	Customer segment, which may be missing

Your question is PySpark Data Cleaning and EDA. Start with the requirements and the two tables in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem