Interview Guides

Design Async LLM Job Platform

Hard

Generative AI & LLMs

Context

BeaconCMS wants an event-driven AI platform for asynchronous tasks such as product-description generation, metadata enrichment, moderation summaries, and taxonomy tagging. Requests come from multiple internal products and must be processed reliably without blocking user-facing APIs.

Constraints

API acknowledgment to producer: <150ms p95
End-to-end completion SLA: 95% of jobs within 2 minutes; 99% within 10 minutes
Cost ceiling: $35K/month at 8M jobs/month
Hallucination ceiling: <2% for grounded enrichment fields on a 1,000-job golden set
Prompt injection success rate: <0.5% on adversarial inputs
PII must not be persisted in prompts, logs, or dead-letter payloads
System must support retries, idempotency, partial failures, and human review for high-risk outputs

Available Resources

Event bus (Kafka or SNS/SQS), object store, Postgres, Redis, and a workflow engine
Input payloads include raw content, product metadata, locale, customer tier, and optional reference documents
Approved LLMs: a low-cost small model for classification/extraction and a higher-quality model for generation
Existing policy service for content safety and a rules engine for deterministic validations
50K historical jobs with human-reviewed outcomes; 1,000 examples can be labeled for a golden set

Task

Design the event-driven architecture for job intake, orchestration, retries, idempotency, state tracking, and result delivery across multiple asynchronous AI tasks.
Specify how prompts, tool calls, and structured outputs differ for generation vs enrichment jobs, including refusal behavior and safety checks.
Define an evaluation-first plan: offline quality/safety evaluation before launch, then online monitoring, alerting, and rollback criteria.
Estimate cost and latency by stage, and explain routing decisions between small vs large models under the monthly budget.
Identify key failure modes such as hallucinated fields, prompt injection in source content, duplicate processing, and stuck jobs, with concrete mitigations.

Design Async LLM Job Platform

Hard

Generative AI & LLMs

Context

Constraints

API acknowledgment to producer: <150ms p95
End-to-end completion SLA: 95% of jobs within 2 minutes; 99% within 10 minutes
Cost ceiling: $35K/month at 8M jobs/month
Hallucination ceiling: <2% for grounded enrichment fields on a 1,000-job golden set
Prompt injection success rate: <0.5% on adversarial inputs
PII must not be persisted in prompts, logs, or dead-letter payloads
System must support retries, idempotency, partial failures, and human review for high-risk outputs

Available Resources

Event bus (Kafka or SNS/SQS), object store, Postgres, Redis, and a workflow engine
Input payloads include raw content, product metadata, locale, customer tier, and optional reference documents
Approved LLMs: a low-cost small model for classification/extraction and a higher-quality model for generation
Existing policy service for content safety and a rules engine for deterministic validations
50K historical jobs with human-reviewed outcomes; 1,000 examples can be labeled for a golden set

Task

Design the event-driven architecture for job intake, orchestration, retries, idempotency, state tracking, and result delivery across multiple asynchronous AI tasks.
Specify how prompts, tool calls, and structured outputs differ for generation vs enrichment jobs, including refusal behavior and safety checks.
Define an evaluation-first plan: offline quality/safety evaluation before launch, then online monitoring, alerting, and rollback criteria.
Estimate cost and latency by stage, and explain routing decisions between small vs large models under the monthly budget.
Identify key failure modes such as hallucinated fields, prompt injection in source content, duplicate processing, and stuck jobs, with concrete mitigations.

Your Answer

Design Async LLM Job Platform

Hard

Generative AI & LLMs

Context

Constraints

API acknowledgment to producer: <150ms p95
End-to-end completion SLA: 95% of jobs within 2 minutes; 99% within 10 minutes
Cost ceiling: $35K/month at 8M jobs/month
Hallucination ceiling: <2% for grounded enrichment fields on a 1,000-job golden set
Prompt injection success rate: <0.5% on adversarial inputs
PII must not be persisted in prompts, logs, or dead-letter payloads
System must support retries, idempotency, partial failures, and human review for high-risk outputs

Available Resources

Event bus (Kafka or SNS/SQS), object store, Postgres, Redis, and a workflow engine
Input payloads include raw content, product metadata, locale, customer tier, and optional reference documents
Approved LLMs: a low-cost small model for classification/extraction and a higher-quality model for generation
Existing policy service for content safety and a rules engine for deterministic validations
50K historical jobs with human-reviewed outcomes; 1,000 examples can be labeled for a golden set

Task

Design the event-driven architecture for job intake, orchestration, retries, idempotency, state tracking, and result delivery across multiple asynchronous AI tasks.
Specify how prompts, tool calls, and structured outputs differ for generation vs enrichment jobs, including refusal behavior and safety checks.
Define an evaluation-first plan: offline quality/safety evaluation before launch, then online monitoring, alerting, and rollback criteria.
Estimate cost and latency by stage, and explain routing decisions between small vs large models under the monthly budget.
Identify key failure modes such as hallucinated fields, prompt injection in source content, duplicate processing, and stuck jobs, with concrete mitigations.

Design Async LLM Job Platform

Hard

Generative AI & LLMs

Context

Constraints

API acknowledgment to producer: <150ms p95
End-to-end completion SLA: 95% of jobs within 2 minutes; 99% within 10 minutes
Cost ceiling: $35K/month at 8M jobs/month
Hallucination ceiling: <2% for grounded enrichment fields on a 1,000-job golden set
Prompt injection success rate: <0.5% on adversarial inputs
PII must not be persisted in prompts, logs, or dead-letter payloads
System must support retries, idempotency, partial failures, and human review for high-risk outputs

Available Resources

Event bus (Kafka or SNS/SQS), object store, Postgres, Redis, and a workflow engine
Input payloads include raw content, product metadata, locale, customer tier, and optional reference documents
Approved LLMs: a low-cost small model for classification/extraction and a higher-quality model for generation
Existing policy service for content safety and a rules engine for deterministic validations
50K historical jobs with human-reviewed outcomes; 1,000 examples can be labeled for a golden set

Task

Design the event-driven architecture for job intake, orchestration, retries, idempotency, state tracking, and result delivery across multiple asynchronous AI tasks.
Specify how prompts, tool calls, and structured outputs differ for generation vs enrichment jobs, including refusal behavior and safety checks.
Define an evaluation-first plan: offline quality/safety evaluation before launch, then online monitoring, alerting, and rollback criteria.
Estimate cost and latency by stage, and explain routing decisions between small vs large models under the monthly budget.
Identify key failure modes such as hallucinated fields, prompt injection in source content, duplicate processing, and stuck jobs, with concrete mitigations.