Benchmarking

Atlas's performance claims are measurable. This chapter shows what the benchmarks measure, how to reproduce them, and how to read the numbers.

The headline numbers

From the repo README, distilled:

And the kernel micro-benchmark summary: Atlas wins 32/32 against PyTorch on attention, GEMM, SSM, RMSNorm, RoPE, SiLU×Mul, and conv1d, with speedups from 1.04× up to 18.2×.

Two kinds of benchmark

Atlas has two benchmark surfaces:

1. End-to-end HTTP throughput — atlas-spark-bench (client-side Criterion harness targeting a running server). This is what "131 tok/s" means.
2. Per-kernel micro-benchmarks — Criterion benches in each primitive crate, run with cargo bench. This is where "4.95× prefill attention" comes from.

Different things; both are meaningful. The E2E number is what an operator sees. The per-kernel number is what tells the kernel engineer where effort is paying back.

Running end-to-end benchmarks

Start a server:

sudo docker run -d --name atlas-35b \
  --network host --gpus all --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  avarok/atlas-gb10:latest \
  serve Sehyo/Qwen3.5-35B-A3B-NVFP4 \
    --max-seq-len 8192 --kv-cache-dtype nvfp4 \
    --scheduling-policy slai \
    --speculative --mtp-quantization nvfp4

Wait for listening. Then:

export ATLAS_BENCH_URL=http://localhost:8888
cd /path/to/atlas
cargo bench -p atlas-spark-bench

Criterion saves results to target/criterion/. The stable JSON snapshots that the README quotes are pinned under bench/.

The scripts/sweep_all_models.sh helper boots each model in turn, runs the canonical short-prompt bench, and writes the README.md throughput table. That's how the table in the README gets regenerated.

Running per-kernel benchmarks

cargo bench -p spark-runtime        # KV cache ops, sampler micro
cargo bench -p atlas-spark-bench    # end-to-end client benchmarks

Criterion-driven, from each crate's benches/*.rs. Reference shapes come from Qwen3-Next-80B (hidden=2048, 16 Q-heads, 2 KV-heads, head_dim=256, intermediate=512, num_experts=256, topk=10).

The full kernel numbers table:

32/32 wins. Peak achieved memory bandwidth in this set: 599 GB/s (2.2× the 273 GB/s LPDDR5X spec — that's the L2 cache effect for small SiLU×Mul inputs).

Concurrency sweep

scripts/run_conc_benchmark.sh drives N parallel streams against one server. Reveals the scheduler + KV allocator under load. Typical pattern on Qwen3.5-35B:

Aggregate throughput grows super-linearly up to the batch size cap (where graph amortisation kicks in) and then super-linearly until KV pool pressure; after that latency degrades more than throughput improves. Sweet spot on 35B: concurrency 4–8.

TTFT and prefix-cache behaviour

Run a request, note the TTFT. Run the same request again — with --enable-prefix-caching, TTFT drops to ~40ms (prefix cache hit). Agent workloads observe this as the difference between "the first response was slow" and "everything after is fast". The bench harness's prefix-warmup suite measures cold vs warm TTFT.

Where raw results live

• Pinned snapshots (tracked): result files under bench/. These feed the book and the README.
• Ephemeral Criterion runs (gitignored): target/criterion/.
• Historical benchmark journeys: docs/ATLAS_SPARK_JOURNEY.md — the benchmark retrospective across the Spark line.

Apples-to-apples notes

When comparing Atlas to vLLM or TensorRT-LLM:

• Same hardware. GB10 SM121 numbers do not transfer to H100 / B200 / MI300X.
• Same model. "Qwen3.5-35B-A3B at 36 tok/s" is vLLM's NVIDIA GB10 benchmark on the NVFP4 CUTLASS MoE path, same HF checkpoint.
• Same prompt shape. The 131 tok/s number is on a short prompt ("What is the capital of France?", max_tokens ≤ 30). Longer prompts show slightly different numbers because prefill cost amortizes differently.
• Same precision. Atlas NVFP4 vs vLLM NVFP4; Atlas FP8 vs vLLM FP8. Never compare across quant schemes.

The headline "3.6× faster than NVIDIA's 36 tok/s" is apples-to-apples against NVIDIA's own vLLM numbers on the same (GB10, Qwen3.5-35B-A3B, NVFP4) target.

Files to read

From the CLI

The benchmark suite the dashboard runs is also a subcommand, so a benchmark can be scripted, run in CI, or driven over SSH with no terminal attached.

spark benchmark list                      # the suite
spark benchmark list concurrency-sweep    # one benchmark's parameters
spark benchmark run  concurrency-sweep --model <served-model>
spark benchmark history

run drives an endpoint that is already serving — it neither loads a model nor touches the GPU. The one exception is --pull-request-gate, which does start a server: it serves the benchmark's own recipe on a free port (900 s boot timeout, for a cold NVFP4 load) and tears it down on drop (cli/bench_selfstart.rs).

A benchmark can be defined on more than one model variant — one BENCH.toml entry per checkpoint, each carrying its own serve recipe and its own thresholds (spark benchmark list <id> prints them, and the TUI shows them as a step after selecting the benchmark). A gate run serves the variant marked default = true unless you name another:

spark benchmark run agentic-webserver --yes --pull-request-gate   --checkpoint unsloth/Qwen3.8-27B-NVFP4

Records are keyed by variant too — a non-default variant's record gets the checkpoint slug in its filename, and only the default variant's records can discharge the required gate. Numbers never compare across variants: the dense 27B's wall band is roughly 2× the 35B MoE's, which is exactly why the thresholds live per checkpoint.

Point run somewhere else with --url:

spark benchmark run concurrency-sweep \
  --url http://10.10.10.3:8888 --model Qwen/Qwen3.6-35B-A3B-FP8 \
  --param concurrencies=1,2,4 --param isls=128 --param osl=64

--param takes any key from spark benchmark list <id>; anything you leave out takes the schema default. An unknown key is an error listing the valid ones, because a silently-ignored override produces a run measuring something other than what you asked for.

Sharded benchmarks (known-answer tests)

Two legs dominate every certification campaign: the BFCL draws are ~1000 samples each and run for hours, so a failure in one is not visible until the campaign is nearly over. They are known-answer tests — an input, an expected output, scored for accuracy — so the work is embarrassingly parallel: split the draw, run the pieces on different boxes, merge, score once.

bfcl-subset and bfcl-subset-echolp are therefore benchmark groups. The gate id is unchanged; what changed is how its number is produced — and since 2026-09-13 a complete partition of shards is the only thing that produces it: a whole-draw record under the gate's own id no longer satisfies the gate, and the verdict says so by name if one is all the directory holds. A shard is the group's own benchmark run with --param shard=i/n; its record is filed under the group with -s<i>of<n> in the name and carries shard.index / shard.count in its metrics. The shard count is not fixed: the campaign picks n for the fleet it has — spark bench certify defaults to two shards per box, and one box alone runs the whole draw as 0/1 — and the verdict accepts the newest complete partition the records at one commit form, whatever its n. A partition begun at this commit is finished at its own count (gate::shards_owed), never restarted at another.

spark benchmark run bfcl-subset --pull-request-gate --hardware gb10 --param shard=0/2   # on dgx1
spark benchmark run bfcl-subset --pull-request-gate --hardware gb10 --param shard=1/2   # on dgx2
spark benchmark aggregate bfcl-subset      # what the group scores, and what is missing
spark bench certify --shards 6             # or let the campaign choose and place them

Selection is a stride within each subset — row i goes to shard i % n — so every shard gets a proportional slice of every subset, and a 16-row subset does not vanish from most of them.

The index is 0-based, so the whole draw is 0/1 and 1/1 is refused (it is index 1 of one shard). Run by hand without --param shard, the benchmark measures the whole draw and writes a whole-draw record — fine for a measurement, not evidence for the gate.

What the group refuses, and why

Merging counts is only sound if the parts really are the draw, so a group is judged only when these conditions hold. Each of these was a way to get a passing number for a measurement that never happened:

The second deserves emphasis: the samples threshold is pinned exactly (min == max == 995) and cannot catch a duplicated shard, because the duplicate still contributes the right number of rows.

Aggregation is over counts, never scores. score.py weights hierarchically, so the mean of shard scores is not the whole-set value; the group sums each subset's (hits, n) integers and applies the hierarchy once.

The number is partition-dependent, and that is the certified regime

The shards are scored open: cross-request SSM snapshot reuse stays on, as in production, and the serve is not --hermetic. The consequence is measured (#936): running the golden draw whole and as its four shards at one commit changes the answer on 12 of 995 samples — ten in live_irrelevance, one each in live_multiple and live_parallel_multiple — because a request restores from whichever SSM snapshot an earlier request left behind. The twelve are listed in benchmarks::bfcl::sensitive; every run warns on each one it scores and reports the count as known_partition_sensitive. The floors for both gates are cut from the sharded aggregate, so the bar and the measurement are taken under the same regime. --hermetic closes the channels (0 of 995) and is the subject of kat-equality-gate, not of these gates.

Only Sensitivity::Correctness benchmarks may be grouped. For a speed benchmark the timing is the number, and four quarter-length runs across three boxes have a different wall, TTFT distribution and concurrency profile from one serial run — no arithmetic recovers the original.

Exit codes

1 and 2 are distinct so a script can tell "the harness broke" from "the model missed the bar". --no-fail-on-verdict collapses 2 into 0 when you are collecting numbers rather than gating on them.

Run history

Every run — from the CLI or the dashboard — is recorded under ~/.atlas/runs/<benchmark-id>/, carrying the result, every parameter used (not just the ones you overrode), the target, the source, and the Atlas version. So a stored run says what it measured and can be reproduced.

spark benchmark history --id concurrency-sweep --limit 5
spark benchmark history --run run-1785000000123456789 --format json | jq .params

One store, both directions: a CLI run appears in the dashboard's Benchmarks → History pane, and a dashboard run appears in spark benchmark history marked tui.

Machine-readable output goes to stdout, progress to stderr, so --format json > run.json is a clean file. ATLAS_HOME relocates the store.

• crates/atlas-spark-bench/src/lib.rs — E2E harness.
• Each primitive crate's benches/*.rs — per-kernel micro.
• bench/*.json — pinned result snapshots.
• scripts/sweep_all_models.sh, scripts/run_conc_benchmark.sh — automation.
• docs/ATLAS_SPARK_JOURNEY.md — benchmark journey and retrospective.
• README "Benchmark Results" section — the authoritative long-form table.