Structured tasks
Classifiers are evaluated with exact labels, invalid output rates, and output contract adherence.
Workflow scorecards
Benchmarked by legal task, not generic leaderboard score.
QLLM reports task-specific accuracy, faithfulness, citation behavior, format adherence, latency, and cost. Open-ended legal outputs require lawyer review before external claims.
Latency caveat: the current live measurements are cold-start staging results through Modal-hosted backends. They are not warmed steady-state SLA numbers.
Initial live benchmark results
| Area | N | Result | Cold-start p95 |
|---|---|---|---|
| Compliance concept detection | 2 | 1.000 accuracy | 155s |
| Clause diff classification | 2 | 0.500 accuracy | 180s |
| Contract type classification | 2 | 0.500 accuracy | 2.3s |
| RAG no-source refusal | 2 | 0.500 refusal behavior | 0.1s |
| Regular 70B summarization | 3 | 0.767 faithfulness precheck | 442s |
RAG and summaries
Correctness-sensitive answers are scored against source context and queued for lawyer review.
Latency and cost
Cold-start and warm-start runs are tracked separately so commercial SLAs are not overstated.