Running Consensus From Annotation Events | Dataford Interview Questions

Given a stream of real-time annotation events, write an efficient algorithm to calculate the running consensus score among multiple annotators for HumanSignal.

Each event is a dictionary with task_id, annotator_id, and label. A new event replaces that annotator's previous label for the same task. Return the affected task's consensus score after every event, where the score is the frequency of the most common current label divided by the number of annotators who have labeled that task.

Use def running_consensus(events):. Return a list of dictionaries containing task_id and score in event order.

Constraints

0 <= events.length <= 9999
Each event contains task_id, annotator_id, and label
task_id, annotator_id, and label are hashable values
An annotator has at most one current label per task
Each event replaces the annotator's previous label for the same task

Function Signature

def running_consensus(events):

Your question is Running Consensus From Annotation Events. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.

Problem

Given a stream of real-time annotation events, write an efficient algorithm to calculate the running consensus score among multiple annotators for HumanSignal.

Use def running_consensus(events):. Return a list of dictionaries containing task_id and score in event order.

Constraints

0 <= events.length <= 9999
Each event contains task_id, annotator_id, and label
task_id, annotator_id, and label are hashable values
An annotator has at most one current label per task
Each event replaces the annotator's previous label for the same task

Function Signature

def running_consensus(events):

Your question is Running Consensus From Annotation Events. Start with the requirements in the Question tab.

Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.