Analyze Self-Attention for Feed Ranking

Business Context

Meta wants to improve lightweight text understanding for Facebook Feed ranking features derived from post text and comment context. You are asked to explain self-attention mathematically, implement it correctly, and reason about whether it can meet production latency constraints.

Dataset

You are given a supervised learning dataset built from public-post text embeddings and engagement labels used for an offline ranking proxy.

• Size: 2.4M examples, max sequence length 256, vocabulary size 50K
• Target: Binary — high engagement in the next 24 hours (1) vs not high engagement (0)
• Class balance: 18% positive, 82% negative
• Missing data: ~6% missing in metadata features; text is always present but sequence lengths vary heavily

Success Criteria

A strong solution should:

• derive the self-attention equations clearly, including query, key, value projections and softmax normalization
• explain why scaled dot-product attention uses the factor $1/\sqrt{d_k}$
• quantify time and memory complexity with respect to sequence length and hidden size
• implement a correct attention module and train a small classifier baseline
• discuss when self-attention becomes impractical for long Facebook Feed sequences and what approximations are reasonable

Constraints

• P95 online inference budget for the text encoder is under 20 ms on a single production GPU batch
• The solution should support variable-length sequences with padding masks
• Researchers care about correctness and scaling behavior more than leaderboard performance

Deliverables

1. Derive scaled dot-product self-attention mathematically, including tensor dimensions.
2. Analyze computational and memory complexity for single-head and multi-head attention.
3. Implement a PyTorch attention-based classifier with masking.
4. Evaluate against a mean-pooled embedding baseline.
5. Recommend whether this architecture is suitable for Facebook Feed ranking under the stated latency constraints.