LLM Ticket Classification and Routing

How do you approach fine-tuning or prompt-engineering an LLM to automatically classify and route incoming user support tickets?

Using the provided ticket, prediction, and queue data, write a query to evaluate which model versions and predicted queues are reliable for automatic routing. Include only predictions with confidence of at least 0.70 and tickets with a known actual queue.

Output

1. One row per model version and predicted queue
2. Columns: model_version, predicted_queue, eligible_tickets, correct_tickets, accuracy_pct, and avg_confidence
3. Sort by model_version, accuracy descending, then predicted queue alphabetically