Your question is Attention-Free Transformer in PyTorch. Start with the requirements and the two tables on the right.
Run and submit as often as you like. When you're ready, talk me through your approach or go straight to the code.
Implement a custom Attention-Free Transformer module in PyTorch based on a provided formula (where Q is used as a gating mechanism rather than query, K and a trainable positional embedding act as logits for softmax).
Asked in the Technical Coding Round stage. AI autocomplete tools were disabled, and the code did not need to run fully, but the logic and structure needed to look reasonable. Reported follow-ups: how would you extend this implementation to multi-head, and how can you debottleneck this and reduce the computational complexity to linear, similar to Linear Attention?
Return the module output tensor shape and the per-layer parameter audit in the required order.
| Column | Type | Description |
|---|---|---|
| layer_idPK | INT | Unique layer identifier |
| layer_name | VARCHAR(100) | Transformer layer name |
| parameter_name | VARCHAR(100) | Parameter name within the layer |
| is_trainable | BOOLEAN | Whether the parameter is trainable |
| Column | Type | Description |
|---|---|---|
| output_idPK | INT | Unique output record identifier |
| module_name | VARCHAR(100) | Module name |
| layer_name | VARCHAR(100) | Layer name referenced by the output |
| output_shape | VARCHAR(50) | Tensor output shape |