Your question is Knowledge Distillation for Transformers. Take a moment with it on the right.
Talk me through your thinking if you like. When you're confident, submit your answer and I'll grade it like a real screen (7/10 or better passes).
Walk me through how you would implement knowledge distillation to train a smaller student model from a large transformer.
Explain the teacher and student architectures, temperature-scaled soft targets, hard-label supervision, loss weighting, data preparation, optimization, and evaluation. Provide production-quality Python or PyTorch code that demonstrates the training loop and explain how you would preserve task quality while reducing latency, memory, and inference cost. Address teacher calibration, sequence-level versus token-level distillation, variable sequence lengths, checkpointing, validation, and failure modes such as overfitting teacher errors or excessive capacity reduction.