Optimizing for 10x Data Growth | Dataford Interview Questions

Your question is Optimizing for 10x Data Growth. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Problem

Cirium aggregates flight-status records from airlines worldwide to compute delay statistics per airport. Below are two functions from that pipeline, written when the dataset was much smaller.

case class FlightRecord(flightNumber: String, airport: String, delayMinutes: Int)

def averageDelayByAirport(records: List[FlightRecord]): Map[String, Double] = {
  val airports = records.map(_.airport).distinct

  var result = Map[String, Double]()
  for (airport <- airports) {
    val matching = records.filter(_.airport == airport)
    val total = matching.map(_.delayMinutes).sum
    result = result + (airport -> total.toDouble / matching.size)
  }

  result
}

def worstFlights(records: List[FlightRecord]): List[FlightRecord] = {
  records.sortBy(_.delayMinutes).reverse.take(20)
}

How would you optimize this function to handle a 10x increase in data volume at Cirium? Walk through where the cost actually comes from today and what you'd change.

Your question is Optimizing for 10x Data Growth. Take a moment with it on the right.

Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).

Cirium aggregates flight-status records from airlines worldwide to compute delay statistics per airport. Below are two functions from that pipeline, written when the dataset was much smaller.

case class FlightRecord(flightNumber: String, airport: String, delayMinutes: Int)

def averageDelayByAirport(records: List[FlightRecord]): Map[String, Double] = {
  val airports = records.map(_.airport).distinct

  var result = Map[String, Double]()
  for (airport <- airports) {
    val matching = records.filter(_.airport == airport)
    val total = matching.map(_.delayMinutes).sum
    result = result + (airport -> total.toDouble / matching.size)
  }

  result
}

def worstFlights(records: List[FlightRecord]): List[FlightRecord] = {
  records.sortBy(_.delayMinutes).reverse.take(20)
}

How would you optimize this function to handle a 10x increase in data volume at Cirium? Walk through where the cost actually comes from today and what you'd change.