Researchers at Carnegie Mellon University and the University of Southern California released ServerBench, a benchmark that evaluates coding agents' ability to design large language model serving systems, in a preprint on arXiv this month.
ServerBench assesses agents on a 'dollars per dollar' objective — latency-qualified token value per dollar of GPU cost — under real workload conditions, according to the paper ServerBench paper1. The benchmark requires agents to build a complete serving system from a minimal starter, making decisions about the scheduler, KV cache, batching, kernels, and the number of concurrent sessions. Stateful replay preserves session dependencies from recorded coding-agent sessions, and an independent verification step checks that the submitted system meets latency targets and produces correct outputs.
Existing benchmarks test individual kernels or code patches under fixed loads, while ServerBench scores complete servers on value per GPU dollar under replayed coding-agent traffic, the paper's introduction states ServerBench paper2. The authors argue that prior benchmarks oversimplify workloads, focus on speedup-only objectives, and validate only for correctness.
Across 11 coding-agent models and three model-hardware deployments on L40 GPUs, Claude Sonnet 5.5 achieved 2.97 dollars per dollar on Gemma E4B, or 4.87 times the vLLM baseline, according to the paper. The GitHub repository lists the deployments and results: Gemma E4B on a single L40 GPU reached 4.87 times the strongest baseline (vLLM at 0.61 dollars per dollar), Qwen3.5 9B on two L40 GPUs reached 2.72 times the strongest baseline (SGLang at 1.62 dollars per dollar), and Qwen3.8 27B INT4 on two L40 GPUs reached 1.80 times the strongest baseline (vLLM at 1.47 dollars per dollar) GitHub repository3. Of 33 total runs, 22 yielded systems that passed verification.
The scoring mechanism includes a latency gate: only requests that meet both time-to-first-token and time-per-output-token targets count toward the score. The system is rebuilt fresh, replayed on held-out seeds, and checked for prompt completeness, output lengths, streaming consistency, fidelity to the checkpoint, and answer accuracy; only verified systems receive a score, the paper states ServerBench paper2.
The paper is a preprint with no stated peer review or independent replication. All results are the authors' own, run on their own harness. The benchmark conditions include an 8-hour development budget per run, with agents running through Claude Code or Codex, and a 1,800-second final replay on a held-out workload seed.
The authors identify three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization.