Build a Multimodal LLM for Real-Time Video Editing

Welcome to your interview.

The question is on your right: Build a Multimodal LLM for Real-Time Video Editing. Take a moment with it first.

Talk your thinking through with me if you like - when you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes). Discussion and graded submissions share your five interviewer interactions, so spend them well.

Business Context

VideoPro, a leading video editing software provider with over 1 million active users, aims to integrate AI-driven features that enhance user creativity and efficiency. The goal is to build a multimodal large language model (LLM) that can understand user commands in natural language and interact with video content in real-time, enabling seamless editing workflows.

Dataset

Feature Group	Count	Examples
Video Metadata	10	duration, resolution, frame_rate, codec
User Commands	5	'Trim video', 'Add filter', 'Speed up', 'Add text overlay'
Video Content	50K	frames, audio segments, color histograms, scene descriptors
User Interaction	15	click events, time spent on each tool, undo actions

Size: 50K video clips, each with 100 frames, 4 audio channels, and user interaction logs
Target: Multimodal output based on user commands (e.g., video edits, effects)
Class balance: Varied; some commands are more frequent than others (e.g., 'Trim video' is used 40% of the time, while 'Add text overlay' is used 10% of the time)
Missing data: 5% of user commands missing for new users; 10% of video frames may be corrupted in legacy videos

Requirements

Develop and train a multimodal LLM to process and generate video editing commands.
Achieve an accuracy of 85% in command interpretation and execution.
Ensure the model can process inputs with minimal latency (under 200ms).
Implement a mechanism for real-time user feedback to improve model performance.

Constraints

The model must be capable of real-time inference to support user interactions without noticeable delays.
It should handle various video formats and resolutions, ensuring broad compatibility.
The model should be easily interpretable for debugging and user feedback purposes.