Benchmarking LLM Serving Stacks: Realistic Loads and Production Patterns
Most teams run a quick smoke test on their new Large Language Model server, see green lights in the dashboard, and ship it to production. Then, three weeks later, the first real traffic spike hits, and the system chokes. Why? Because they tested with synthetic data that looked nothing like how humans actually type. Benchmarking an inference stack designed to process natural language queries at scale is not just about seeing if the GPU works. It is about predicting how your infrastructure behaves when ten thousand users ask questions of varying lengths at the exact same second.
The gap between lab results and production reality is where budgets go to die. If you are managing deployment infra, you need a methodology that bridges this gap. This guide breaks down how to build a benchmarking suite that reflects actual user behavior, using specific metrics and tools that have proven effective in high-stakes environments.
Why Standard Benchmarks Fail in Production
Traditional software testing relies on fixed inputs and expected outputs. LLMs break this model. A request for "What is 2+2?" consumes almost no compute resources compared to a request asking for a full legal contract summary based on a 50-page PDF. If your benchmark only uses short prompts, you are measuring the speed of a sprinter, not a marathon runner.
The core issue is variance. In production, input token counts vary wildly. Output lengths depend on stop conditions that can trigger early or run to the maximum limit. To get accurate numbers, you must move beyond simple Requests Per Second (RPS) counts. You need to look at distribution. Specifically, you should track the P50 and P90 latencies rather than averages. An average hides the pain points. If your P90 Time-to-First-Token (TTFT) is five seconds, one out of every ten users feels the lag, even if the average looks healthy.
- Input Variance: Simulate prompt lengths ranging from 10 tokens to 4,096 tokens.
- Output Variance: Force some requests to generate long chains of thought and others to answer concisely.
- Concurrency Spikes: Don't ramp up linearly. Inject sudden bursts of traffic to test autoscaling triggers.
Key Metrics That Actually Matter
When comparing different serving stacks, such as vLLM, TensorRT-LLM, or SGLang, you need a consistent set of KPIs. Here are the four non-negotiables:
| Metric | Definition | Why It Matters |
|---|---|---|
| Time-to-First-Token (TTFT) | Latency from request arrival to the first generated token. | Critical for user perception. High TTFT makes the UI feel broken. |
| Inter-Token Latency (ITL) | Average time between subsequent tokens. | Determines reading speed. Should remain stable during generation. |
| Throughput (Tokens/sec) | Total tokens processed per second across all concurrent requests. | Measures hardware efficiency and batching capability. |
| Cost per Million Tokens | Infrastructure cost divided by total output volume. | The ultimate business metric for ROI. |
Notice what is missing from that list: raw CPU usage. While useful for debugging, CPU percentage tells you little about user experience. Focus on token-level metrics. They correlate directly with the value delivered to the end-user.
Designing Realistic Load Profiles
To create a realistic load profile, you need to mirror your actual traffic mix. Start by pulling anonymized logs from your current production environment. Analyze the distribution of prompt lengths and response lengths. If 80% of your users ask short questions but 20% upload large documents, your benchmark must reflect that 80/20 split.
Use open-source tools like Locust or custom Python scripts with AsyncIO to drive this traffic. Avoid simple HTTP clients that serialize requests. You need true concurrency. For example, if you are testing a model served on an NVIDIA H100, you might configure your load generator to maintain 50 concurrent connections, with each connection sending a new request immediately after receiving the previous response. This keeps the pipeline saturated, which is how modern continuous batching engines operate.
A common mistake is ignoring the network hop. If your benchmark runs on the same machine as the GPU server, you eliminate network latency. This is great for isolating hardware performance, but bad for predicting real-world behavior. Run your load generator from a separate instance in the same availability zone. This adds realistic network jitter and bandwidth constraints without introducing cross-region noise.
Comparing Serving Stacks: A Practical Approach
Let’s look at a concrete scenario. You are evaluating two stacks: Stack A (a standard OpenAI-compatible API wrapper) and Stack B (vLLM with PagedAttention enabled). Both serve the same 70B parameter model on identical A100 GPUs.
In a static benchmark with short prompts, both stacks perform similarly. However, when you introduce long-context workloads (prompts exceeding 8k tokens), the differences emerge. Stack B’s memory management allows it to handle larger batches without swapping to CPU RAM. The result? Stack B maintains a TTFT under 300ms at 50 concurrent users, while Stack A degrades to over 1.2 seconds due to memory pressure.
This is why context length matters more than raw speed. When benchmarking, always include a "long-tail" test case. Take your longest historical prompt, duplicate it 10 times, and fire them simultaneously. Watch how the queue depth grows. If the queue depth spikes and doesn't recover, your serving stack has a bottleneck in its scheduler or memory allocator.
Optimizing for Cost Efficiency
Performance is only half the equation. The other half is cost. A stack that is 10% faster but requires 30% more VRAM might be the wrong choice if you are running on spot instances that frequently evict pods when memory limits are hit.
Calculate the "Efficiency Ratio": Throughput divided by Power Consumption. Tools like NVIDIA DCGM can export power draw data in real-time. Correlate this with your token throughput. You will often find that pushing a GPU to 100% utilization yields diminishing returns in tokens per watt. There is usually a sweet spot around 80-90% utilization where thermal throttling hasn't kicked in, but the hardware is fully engaged.
Also, consider the impact of quantization. Running a model in FP8 instead of FP16 can double your throughput. But does it degrade quality enough to matter? Your benchmark should include a small subset of evaluation tasks (like MMLU or HumanEval) to ensure that the speed gain doesn't come at the cost of accuracy. If FP8 drops your code generation score by 5%, that speed gain might not be worth the support tickets.
Building a Continuous Benchmarking Pipeline
One-off benchmarks are snapshots. They become obsolete the moment you update your Docker image or change your CUDA version. Treat benchmarking as part of your CI/CD pipeline. Every time a new container image is built, run a standardized micro-benchmark suite.
Keep these tests fast. A full production-scale load test takes hours. Instead, run a "canary" benchmark: 100 requests, mixed lengths, 10 concurrent users. If the P90 TTFT deviates by more than 10% from the baseline, fail the build. This catches configuration errors, driver incompatibilities, or silent regressions before they reach staging.
Store these results in a time-series database like Prometheus. Over time, you will see trends. Maybe a new GPU driver improves ITL by 5%. Maybe a library update introduces a memory leak that slowly increases TTFT over 24 hours. These insights are impossible to get from manual testing.
Frequently Asked Questions
What is the ideal concurrency level for benchmarking?
There is no single number. It depends on your model size and hardware. A good starting point is to match your peak production concurrency. If you normally handle 50 concurrent users, test at 50, 100, and 150 to see where the inflection point lies. Look for the point where adding more users stops increasing total throughput.
Should I benchmark on the same machine as the server?
For pure hardware comparison, yes. Running the client on the server eliminates network variables. For production prediction, no. Run the client from a separate node to capture network latency and serialization overhead. Do both: use local runs to isolate GPU bottlenecks and remote runs to validate end-to-end performance.
How do I handle variable output lengths in my metrics?
Do not average total latency across requests of different lengths. Normalize your metrics by token count. Calculate Inter-Token Latency (ITL) separately for each request, then aggregate those ITL values. This ensures that a long, slow response doesn't skew the average for short, fast responses.
Is synthetic data good enough for initial benchmarks?
Yes, provided it mimics the statistical distribution of your real traffic. Random text is useless because it lacks semantic structure that affects attention mechanisms. Use real prompts from your logs, anonymize them, and replay them. If you don't have logs yet, use standard datasets like ShareGPT or Alpaca, but filter them to match your expected context window sizes.
How often should I re-run production benchmarks?
Automate a lightweight version daily. Run a full-scale stress test monthly or whenever you make significant infrastructure changes (e.g., upgrading GPU drivers, changing batch sizes, or deploying a new model version). Consistency is key; if you change the benchmark script, invalidate all previous comparisons.
- Aug, 24 2026
- Collin Pace
- 0
- Permalink
Written by Collin Pace
View all posts by: Collin Pace