/ \
/ \
/ \
System Architecture & /__________\ Advanced SQL & Database
Distributed Compute Internals
### Distributed Computing & PySpark Optimization
This evaluation area tests your understanding of execution mechanics in big data compute frameworks. Interviewers evaluate how well you configure cluster hardware, diagnose performance degradation, and execute complex transformation pipelines.
**Be ready to go over:**
- **Spark Execution Model** – Understanding drivers, executors, cores, stages, tasks, transformations vs. actions, and lineage graphs (DAGs).
- **Optimization Techniques** – Broadcast joins, map-side combines, broadcast variables, and handling data skew using key salting.
- **Partitioning Strategies** – Mechanics of `coalesce` vs. `repartition`, shuffle operations, dynamic partition pruning, and file output layouts.
- **Advanced concepts (less common)** – Registering custom UDFs with Kryo serialization, managing Databricks cluster cost optimization, and configuring Delta Lake ACID transaction properties and `MERGE` upserts.
**Example questions or scenarios:**
- "You have a 100 GB input dataset on AWS S3 consisting of small JSON files. How do you size a PySpark cluster and structure your job to optimize read performance and prevent memory pressure?"
- "Explain step-by-step how PySpark handles a join between a massive 1 TB facts DataFrame and a skewed 10 GB dimension DataFrame."
### Advanced SQL & Database Internals
SQL capabilities are evaluated using complex data manipulation tasks. You are expected to demonstrate familiarity with window functions, recursive queries, dynamic pivots, and legacy database structures such as DB2 or enterprise PostgreSQL.
**Be ready to go over:**
- **Window Functions** – Using `ROW_NUMBER()`, `RANK()`, `DENSE_RANK()`, `LEAD()`, `LAG()`, and `SUM() OVER(PARTITION BY ... ORDER BY ... ROWS BETWEEN ...)`.
- **Recursive CTEs** – Traversing organizational charts, parent-child manager hierarchies, and financial transaction chains.
- **Data Warehousing Patterns** – Designing Dimensional Schemas, Star/Snowflake schemas, and Slowly Changing Dimensions (SCD Type 1, 2, and 3).
- **Advanced concepts (less common)** – Analyzing DB2 query access paths, optimizing static vs. dynamic SQL workloads, and evaluating index structures like B-Tree and Bitmap.
**Example questions or scenarios:**
- "Write a query over a bank transaction table to calculate the 7-day moving average transaction amount for every customer account."
- "How do you design a robust SCD Type 2 merge pattern in PySpark and SQL to keep track of historical updates in customer credit profiles?"
### System Architecture & ETL Engineering
System architecture panels evaluate your ability to design scalable, resilient, and enterprise-grade data platforms. Interviewers look for clear system boundaries, error recovery mechanisms, and secure ingestion patterns.
**Be ready to go over:**
- **Streaming vs. Batch** – Designing hybrid Lambda or Kappa architectures integrating Apache Kafka, Spark Structured Streaming, and Delta Lake.
- **Pipeline Fault Tolerance** – Handling source schema drift, managing idempotent writes, dead-letter queues, and automated retry mechanisms.
- **Cloud Infrastructure** – Infrastructure as Code using Terraform, provisioning compute nodes, securing S3 storage buckets, and configuring IAM access controls.
- **Advanced concepts (less common)** – Mainframe system integrations using JCL, DB2, and IDAA (IBM DB2 Analytics Accelerator) alongside modern AWS ecosystems.
**Example questions or scenarios:**
- "Design an end-to-end data pipeline that ingests continuous real-time credit card transactions from Kafka alongside daily batch account balances from an RDBMS into an analytics platform."
- "A nightly ETL pipeline fails 80% through execution due to an unexpected null constraint in an upstream payload. How do you design the execution flow to prevent partial data loading and ensure idempotency?"