Mobile IntelliSense Under High Latency

Context

NovaIDE is building an AI IntelliSense feature for its mobile coding app. Users expect inline code completions, short explanations, and next-action suggestions even on unreliable cellular networks where round-trip latency is often high.

Constraints

• p95 time to first suggestion: 250ms for local suggestions, 1,500ms when cloud fallback is used
• Cost ceiling: $0.80 per 1,000 sessions, with 2.5 suggestions per session on average
• Acceptance rate of top suggestion: at least 28%
• Hallucination ceiling: fewer than 2% of accepted suggestions should introduce syntax errors or reference unavailable symbols
• Must degrade gracefully offline or on 3G-like networks
• Must not leak source code from private repos to unauthorized services
• Prompt injection risk exists through comments, README files, and retrieved project docs

Available Resources

• 20M anonymized IDE completion events from desktop and mobile
• 200K permissively licensed code files for pretraining or distillation
• On-device budget: up to 1.2GB model size, 6GB RAM phones at p50, intermittent CPU/GPU acceleration
• Cloud access to a GPT-4-class or Claude-class model for fallback requests
• Project-local context: current file, open tabs, symbol table, recent edits, and optional repo docs
• Telemetry for suggestion shown, accepted, edited-after-accept, and latency by network tier

Task

1. Design the end-to-end IntelliSense architecture, including what runs on-device vs. in the cloud and how you handle degraded connectivity.
2. Specify the prompting strategy for cloud fallback, including how you ground on local context and reduce hallucinated APIs or symbols.
3. Define an evaluation plan before implementation: offline datasets, acceptance-quality metrics, hallucination checks, and online experiment design.
4. Explain whether you would use prompt design only, distillation, or fine-tuning for the on-device model, and justify the choice with cost/latency tradeoffs.
5. Identify the main failure modes, including prompt injection and privacy leakage, and propose mitigations.