Every retained configuration and result

Complete catalogue.

98% agent-driven model benchmarks and optimizations.

Benchmarks

Select any result row to show the configuration's applied optimizations, parameters, placement, notes, and evidence.

Needle 2

Size: 45M
Quant: CQ2-bit

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	Cactus Needle	CPU	TODO	TODO	—	?	—	—	—	—	NA	NA
	Cactus Needle	CPU	TODO	TODO	—	?	—	—	—	—	NA	NA
`nd2t16` Needle 2 CQ2 on sixteen R720 Ivy Bridge physical cores.2 results · 0 techniques · 2 parameters Configuration ID needle2-cq2-cactus2-r720-cpu-t16 Created 2026-08-11T16:54:09Z Notes Planned throughput and BFCL configuration with interleaved memory. Applied optimizations No applied technique is cataloged. Uncommon parameters target.artifact_state not-downloaded — Queue creation did not download the model or runtime. target.model_revision 231364ff02542c4060afb5f4795801e0a09f5107 — Pinned official Cactus Compute Needle 2 repository revision. Node placement r720 — target; CPU; 16 threads; 16 batch threads; affinity 0-15; NUMA interleave; Official BFCL scorer and dataset revisions remain to be pinned. r720 — target; CPU; 16 threads; 16 batch threads; affinity 0-15; NUMA interleave; Sixteen physical-core CPU IDs across both sockets; no CUDA. Results and evidence `needle2-bfclv4-planned-20260811` Task text generation Status PLANNED Measured 2026-08-11T16:54:09Z Statistic mean; 3641 samples Model Needle 2 Backend Cactus Needle Hosts r720 Hardware CPU Evidence needle2/benchmark-queue.json Benchmark notes. Reproduce BFCL v4 single-turn with the official scorer. Pin scorer and dataset revisions before execution. Record overall, well-formed, and per-category scores as benchmark metrics. `needle2-cpu-t16-throughput-planned-20260811` Task text generation Status PLANNED Measured 2026-08-11T16:54:09Z Statistic median; 5 samples Model Needle 2 Backend Cactus Needle Hosts r720 Hardware CPU Evidence needle2/benchmark-queue.json Benchmark notes. Matched sixteen-thread point. Retain every sample, exact JSON, latency, throughput, and peak RSS. started_at is the queue time.
	Cactus Needle	CPU	TODO	TODO	—	?	—	—	—	—	NA	NA
`nd2t1` Needle 2 CQ2 on one R720 Ivy Bridge CPU thread.1 result · 0 techniques · 2 parameters Configuration ID needle2-cq2-cactus2-r720-cpu-t1 Created 2026-08-11T16:54:09Z Notes Planned latency and throughput point on CPU 0 with local memory. Applied optimizations No applied technique is cataloged. Uncommon parameters target.artifact_state not-downloaded — Queue creation did not download the model or runtime. target.model_revision 231364ff02542c4060afb5f4795801e0a09f5107 — Pinned official Cactus Compute Needle 2 repository revision. Node placement r720 — target; CPU; 1 threads; 1 batch threads; affinity 0; NUMA local-node0; CPU-only run; CUDA devices must not be visible. Results and evidence `needle2-cpu-t1-throughput-planned-20260811` Task text generation Status PLANNED Measured 2026-08-11T16:54:09Z Statistic median; 5 samples Model Needle 2 Backend Cactus Needle Hosts r720 Hardware CPU Evidence needle2/benchmark-queue.json Benchmark notes. Five fixed-payload samples after one warmup. Retain exact JSON, prefill and decode speed, request latency, and peak RSS. started_at is the queue creation time.

TinyLlama OpenOrca 1.1B

Size: 1B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	CPU + P40	1024	25	—	✕	—	—	1017.3	164.3	NA	NA
`tlp2` TinyLlama server on the remote P40 through llama.cpp RPC.1 result · 0 techniques · 4 parameters Configuration ID tinyllama-llamab64739e-p40-rpc-server-b128-c1k-f16 Created 2026-08-11T02:15:43Z Notes Matched remote accelerator control through TCP on roce0. Applied optimizations No applied technique is cataloged. Uncommon parameters accelerator:r720xd-p40.allocated_vram 810 MiB MiB — Observed after model load. backend.rpc_transport TCP — llama.cpp RPC v4.0.0 on 10.42.0.2:50052. backend.split_mode none — RPC0 only. system.resident_gpu_services stopped — local-ai-router and ComfyUI were stopped for the matched series. Node placement r720 — orchestrator; CPU; 16 threads; 16 batch threads; network roce0; llama-server coordinated RPC0 and retained a 282 MiB CUDA context. r720xd — target; CUDA; Tesla P40; 22 target layers; 22 GPU layers; 20 threads; network roce0; RPC0 held the complete model; rpc-server allocated 810 MiB. Results and evidence `tinyllama-p40-rpc-control-20260811` Task text generation Status VALID Measured 2026-08-11T02:15:43Z Statistic median; 5 samples Model TinyLlama OpenOrca 1.1B Backend llama.cpp Hosts r720,r720xd Hardware Tesla P40 Context window 1024 Tested depth 0 Prompt tokens 25 Generated tokens 64 Prefill 1017.3 tok/s Decode 164.3 tok/s Baseline tlv2 (-18.3%) Output matches baseline yes Evidence distributed/results/20260811T021316Z-tinyllama-p40-rpc-control Benchmark notes. The model tensors were on RPC0, the P40. All five outputs matched the V100 control exactly. Result notes. Decode samples: 164.336849, 169.344873, 154.728403, 152.091255, 171.846681 tok/s.
	llama.cpp/poolside	V100	4096	128	—	✕	—	—	4150.9	384.1	NA	29%
`tlv1` TinyLlama OpenOrca complete CUDA0 placement.1 result · 2 techniques · 1 parameter Configuration ID tinyllama-poolside04b2b72-v100-full-b2048-c4k-f16-fa Created 2026-07-29T23:05:58Z Notes Five llama-bench repetitions with a 1 GiB fit margin. Applied optimizations Flash attention Flash Attention was enabled with F16 KV. GPU layer offload The complete 1.1B model was placed on CUDA0. Uncommon parameters system.gpu_services unloaded — The model router and ComfyUI stayed active but released GPU memory. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 22 target layers; 22 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; CUDA_VISIBLE_DEVICES=0; complete target placement. Results and evidence `tinyllama-v100-20260729` Task text generation Status VALID Measured 2026-07-29T23:05:58Z Statistic median; 5 samples Model TinyLlama OpenOrca 1.1B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 128 Generated tokens 32 Prefill 4150.9 tok/s Decode 384.1 tok/s Evidence results/20260729T231000Z-remaining-models/tinyllama-v100.jsonl Benchmark notes. Complete CUDA0 placement. Five pp128 and five tg32 samples. Result notes. Prefill samples: 3690.35, 3917.81, 4150.89, 4513.37, 4538.32. Decode samples: 383.811, 383.928, 384.134, 385.613, 386.216.
	llama.cpp	V100	1024	25	—	✕	—	—	2547.6	201.3	NA	15%
`tlv2` TinyLlama server on physical V100 GPU 1.1 result · 0 techniques · 3 parameters Configuration ID tinyllama-llamab64739e-v100-server-b128-c1k-f16 Created 2026-08-11T02:15:14Z Notes Matched local accelerator control for the RPC tensor split. Applied optimizations No applied technique is cataloged. Uncommon parameters accelerator:r720-v100-32gb-2.allocated_vram 964 MiB MiB — Observed after model load. backend.split_mode none system.resident_gpu_services stopped — local-ai-router and ComfyUI were stopped for the matched series. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 22 target layers; 22 GPU layers; 16 threads; 16 batch threads; CUDA_VISIBLE_DEVICES=1; complete model placement. Results and evidence `tinyllama-v100-control-20260811` Task text generation Status VALID Measured 2026-08-11T02:15:14Z Statistic median; 5 samples Model TinyLlama OpenOrca 1.1B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 25 Generated tokens 64 Prefill 2547.6 tok/s Decode 201.3 tok/s Evidence distributed/results/20260811T021316Z-tinyllama-v100-control Benchmark notes. One untimed warm request preceded five retained requests. All five outputs had the same SHA-256. Result notes. Decode samples: 203.680896, 201.266722, 199.998750, 205.829477, 196.677381 tok/s.
	llama.cpp	V100 + P40	1024	25	—	✕	—	—	176.1	12.6	NA	0%
`tltp` TinyLlama tensor parallel across V100 and remote P40.1 result · 1 technique · 7 parameters Configuration ID tinyllama-llamab64739e-v100-p40-rpc-tensor-1-1-b128-c1k-f16 Created 2026-08-11T02:16:03Z Notes Experimental split-mode tensor with equal V100 and P40 shares. Applied optimizations Tensor parallelism Experimental tensor mode executed correctly across V100 and P40, but RPC TCP reduced decode from 201.267 to 12.590 tok/s. Uncommon parameters accelerator:r720-v100-32gb-2.allocated_vram 656 MiB MiB — Observed after model load. accelerator:r720xd-p40.allocated_vram 502 MiB MiB — Observed after model load. backend.rpc_transport TCP — llama.cpp RPC v4.0.0 on 10.42.0.2:50052; no NCCL or verbs. backend.split_mode tensor — Experimental llama.cpp tensor meta-device. backend.tensor_split 1,1 — Equal segments on CUDA0 and RPC0. system.resident_gpu_services stopped — local-ai-router and ComfyUI were stopped for the matched series. target.output_sha256 daaa9aade70ef9088d82587e2b980304f06580029859c3d33c3ebb176567e2b6 — Matched every V100 control output. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 22 target layers; 22 GPU layers; 16 threads; 16 batch threads; network roce0; CUDA0 held one equal tensor segment and allocated 656 MiB. r720xd — target; CUDA; Tesla P40; 22 target layers; 22 GPU layers; 20 threads; network roce0; RPC0 held the other tensor segment and allocated 502 MiB. Results and evidence `tinyllama-v100-p40-tp-20260811` Task text generation Status VALID Measured 2026-08-11T02:16:03Z Statistic median; 5 samples Model TinyLlama OpenOrca 1.1B Backend llama.cpp Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Context window 1024 Tested depth 0 Prompt tokens 25 Generated tokens 64 Prefill 176.1 tok/s Decode 12.6 tok/s Baseline tlv2 (-93.7%) Output matches baseline yes Evidence distributed/results/20260811T021316Z-tinyllama-v100-p40-tp-tensor Benchmark notes. Experimental tensor mode split all supported tensors and KV equally across CUDA0 and RPC0. All outputs matched the V100 control. Result notes. Decode samples: 12.397469, 12.697738, 12.589811, 12.642902, 12.480991 tok/s. The median was 16.0 times slower than V100.

Bonsai

Size: 8B
Quant: Q1_0

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	CPU	TODO	128	—	✕	—	—	16.1	6.7	NA	NA
`b8c` Legacy configuration for bonsai8-cpu-t161 result · 2 techniques · 0 parameters Configuration ID bonsai8-cpu-t16 Created 2026-07-20T03:31:42Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations CPU thread tuning The Bonsai CPU benchmark used 16 inference threads and 16 batch threads. NUMA memory policy The CPU benchmark used the runtime NUMA distribution policy. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 0 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `bonsai8-cpu-t16` Task text generation Status VALID Measured 2026-07-20T03:31:42Z Statistic mean; 3 samples Model Bonsai 8B Backend llama.cpp Hosts forge Hardware CPU Prompt tokens 128 Generated tokens 32 Prefill 16.1 tok/s Decode 6.7 tok/s Evidence bonsai/benchmark-results.json Benchmark notes. llama-bench with NUMA distribution, batch 2048, and microbatch 512. Result notes. Mean of three pp128 and tg32 samples.
	llama.cpp	P40	TODO	128	—	✕	—	—	717.2	63.5	NA	21%
`b8p` Legacy configuration for bonsai8-p401 result · 1 technique · 0 parameters Configuration ID bonsai8-p40 Created 2026-07-20T15:46:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations NUMA memory policy The P40 benchmark used the runtime NUMA distribution policy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla P40; 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `bonsai8-p40` Task text generation Status VALID Measured 2026-07-20T15:46:00Z Statistic mean; 3 samples Model Bonsai 8B Backend llama.cpp Hosts r720 Hardware Tesla P40 Prompt tokens 128 Generated tokens 32 Prefill 717.2 tok/s Decode 63.5 tok/s Evidence bonsai/benchmark-p40-results.json Benchmark notes. Full P40 offload with NUMA distribution, batch 2048, and microbatch 512. Result notes. Mean of three pp128 and tg32 samples with full P40 offload.
	llama.cpp	V100	TODO	128	—	✕	—	—	1338.8	123.3	NA	16%
`b8v` Legacy configuration for bonsai8-v1001 result · 3 techniques · 0 parameters Configuration ID bonsai8-v100 Created 2026-07-22T22:22:35Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations GPU layer offload The complete Bonsai 8B target was on the V100. NUMA memory policy The V100 benchmark used the runtime NUMA distribution policy. Weight quantization The target artifact used Q1_0 weights. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `bonsai8-v100` Task text generation Status VALID Measured 2026-07-22T22:22:35Z Statistic mean; 3 samples Model Bonsai 8B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Prompt tokens 128 Generated tokens 32 Prefill 1338.8 tok/s Decode 123.3 tok/s Evidence bonsai/benchmark-v100-results.json Benchmark notes. Full V100 offload with NUMA distribution, batch 2048, and microbatch 512. Result notes. Mean of three pp128 and tg32 samples with full V100 offload.

Gemma 4

Size: 25B / 4B
Quant: UD-Q4_K_XL

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	CPU	4096	20	—	—	—	—	—	10.4	NA	NA
`g26c2` Gemma 4 26B-A4B on forge CPUs with the rebuilt f714bf0 backend.1 result · 3 techniques · 0 parameters Configuration ID gemma26-llamaf714bf0-cpu-t12-c4k-q8-fa-target Created 2026-07-26T09:05:00Z Notes Retained CPU service profile; the backend rebuild alone gave 18.8%. Applied optimizations CPU thread tuning Generation threads 12, 14, and 16 all measured near 10.7 tok/s. NUMA memory policy Interleave with a private no-mmap copy measured 10.678 tok/s against a 10.504 mapped control. This is within run-to-run noise, so the retained profile keeps the simpler mapped load. Runtime selection The backend rebuild from aff6eb6 to f714bf0 gave the complete 18.8% gain. No placement or thread change added to it. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 48 target layers; 0 GPU layers; 12 threads; 16 batch threads; affinity 0-15; NUMA distribute; Rebuilt native Ivy Bridge backend; default mapped load. Results and evidence `gemma26-cpu-f714bf0-20260726` Task text generation Status VALID Measured 2026-07-26T09:05:00Z Statistic median; 5 samples Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Prompt tokens 20 Generated tokens 128 Decode 10.4 tok/s Baseline g26mb (+18.8%) Output matches baseline yes Evidence gemma-xeon/benchmark-f714bf0-20260726.txt Benchmark notes. Rebuilt upstream backend alone gave the gain. Interleave, no-mmap, and 14 or 16 threads were each within noise of this profile. Greedy output matched the prior-build baseline exactly. Result notes. Median of five server requests; 18.8% over the prior-build 8.793 single-request baseline. All samples and the prior-build control produced identical greedy tokens.
	llama.cpp	CPU	TODO	128	—	—	—	—	31.1	9.6	NA	NA
`g26c` Legacy configuration for gemma26-cpu-t121 result · 1 technique · 0 parameters Configuration ID gemma26-cpu-t12 Created 2026-07-19T20:19:31Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations CPU thread tuning The best recorded Gemma 4 26B-A4B CPU point used 12 inference threads. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 0 GPU layers; 12 threads Results and evidence `gemma26-cpu-t12` Task text generation Status VALID Measured 2026-07-19T20:19:31Z Statistic single; 1 sample Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge Hardware CPU Prompt tokens 128 Generated tokens 16 Prefill 31.1 tok/s Decode 9.6 tok/s Evidence gemma-xeon/benchmark-results.txt Benchmark notes. llama-bench thread sweep. Start time estimated from the retained report mtime, confirmed by its Git import commit. Result notes. pp128 and tg16 from llama-bench.
	llama.cpp	CPU	4096	TODO	—	—	—	—	—	8.8	NA	NA
`g26mb` Legacy configuration for gemma26-mtp-baseline1 result · 0 techniques · 0 parameters Configuration ID gemma26-mtp-baseline Created 2026-07-19T07:20:45Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 0 GPU layers; 12 threads; 16 batch threads Results and evidence `gemma26-mtp-baseline` Task text generation Status VALID Measured 2026-07-19T07:20:45Z Statistic single; 1 sample Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Generated tokens 128 Decode 8.8 tok/s Evidence gemma-xeon/benchmark-mtp-local.txt Benchmark notes. Non-speculative baseline from the local MTP width sweep. Start time is from the retained result directory name. Result notes. Matched non-speculative baseline from the MTP width sweep.
	llama.cpp	CPU	4096	20	—	—	—	—	20.8	9.6	NA	NA
`g26rb` Gemma 4 26B forge CPU target-only async-client control.1 result · 0 techniques · 1 parameter Configuration ID gemma26-f714bf0-forge-cpu-asyncclient-t12-c4k-q8 Created 2026-07-28T22:17:52Z Notes Native CPU backend; 12 target threads and 16 batch threads. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_use none — The target client was built without a GPU backend. Node placement forge — target; CPU; 48 target layers; 0 GPU layers; 12 threads; 16 batch threads; affinity 0-15; NUMA distribute; CPU-only target with the native f714bf0 backend. Results and evidence `gemma26-asyncclient-baseline-smoke-20260728` Task text generation Status PROVISIONAL Measured 2026-07-28T22:17:52Z Statistic single; 1 sample Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Tested depth 0 Prompt tokens 20 Generated tokens 32 Prefill 20.8 tok/s Decode 9.6 tok/s Evidence results/20260728-gemma4-26b-remote-draft/smoke-baseline-f714.log Benchmark notes. Matched target-only control for the remote-draft smoke test. Result notes. Twenty prompt tokens and 32 generated tokens.
	llama.cpp	CPU	4096	20	—	Local CPU MTP	+3.9%	57.5%	—	10.8	NA	NA
`g26m2` Gemma 4 26B-A4B with one-token MTP on the rebuilt backend.1 result · 2 techniques · 0 parameters Configuration ID gemma26-llamaf714bf0-cpu-t12-c4k-q8-fa-mtp-w1 Created 2026-07-26T08:45:00Z Notes Not retained: 3.3% gain and greedy divergence from baseline. Applied optimizations Speculative decoding The MTP head proposed one token per target verification step. Speculative policy tuning One proposed token. The 12.9% gain on the older backend fell to about 3 to 4% once the target loop became faster. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target_and_draft; CPU; 48 target layers; 0 GPU layers; 12 threads; 16 batch threads; affinity 0-15; NUMA distribute; Embedded MTP head with a one-token policy on the same backend. Results and evidence `gemma26-mtp-f714bf0-w1-20260726` Task text generation Status VALID Measured 2026-07-26T08:45:00Z Statistic single; 1 sample Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Prompt tokens 20 Generated tokens 128 Decode 10.8 tok/s Baseline g26c2 (+3.3%) Output matches baseline no Evidence gemma-xeon/benchmark-f714bf0-20260726.txt Benchmark notes. MTP gain shrank from 12.9% to about 3 to 4% on the faster rebuilt backend. The token-17 divergence is the known current-upstream behavior, not a change from this rebuild. Result notes. One matched pair on the rebuilt backend: 3.3% over its own 10.504 tok/s control, or 3.9% over the retained five-sample 10.442 median. Acceptance is unchanged at 57.5%. The token-17 divergence matches the known current-upstream behavior.
	llama.cpp	CPU	4096	TODO	—	Local CPU MTP	+12.9%	57.5%	—	9.9	NA	NA
`g26mw` Legacy configuration for gemma26-mtp-w11 result · 1 technique · 0 parameters Configuration ID gemma26-mtp-w1 Created 2026-07-19T07:20:45Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding The MTP head proposed one token per target verification step. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target_and_draft; CPU; 0 GPU layers; 12 threads; 16 batch threads Results and evidence `gemma26-mtp-w1` Task text generation Status PROVISIONAL Measured 2026-07-19T07:20:45Z Statistic single; 1 sample Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Generated tokens 128 Prefill 40.1 tok/s Decode 9.9 tok/s Baseline g26mb (+12.9%) Output matches baseline no Evidence gemma-xeon/benchmark-mtp-local.txt Benchmark notes. The output diverged from the non-speculative baseline. Start time is from the retained result directory name. Result notes. Prefill is the pp128 target baseline with 16 batch threads. The raw speculative prompt timing was not retained. The decode baseline in the same sweep was 8.793 tok/s.
	llama.cpp	CPU	4096	20	—	Remote CPU full-model draft	-25.2%	100.0%	20.4	7.1	NA	NA
`g26rr` Gemma 4 26B forge CPU target with full-model R720 CPU draft.1 result · 2 techniques · 2 parameters Configuration ID gemma26-f714bf0-forge-r720-remote-full-w1-t12-c4k-q8 Created 2026-07-28T22:28:02Z Notes One asynchronous token with lookahead three; native CPU backends. Applied optimizations Draft placement The R720 CPU draft loop was slower than the forge CPU target loop. Speculative decoding The full-model remote draft accepted 16 of 16 proposals but reduced decode throughput by 25.2%. Uncommon parameters draft.lookahead_tokens 3 tokens — The asynchronous draft worker could run three tokens ahead. system.gpu_use none — Both target and draft used CPU-only backends. Node placement forge — target; CPU; 48 target layers; 0 GPU layers; 12 threads; 16 batch threads; affinity 0-15; NUMA distribute; CPU-only target with the native f714bf0 backend. r720 — draft; CPU; 0 GPU layers; 12 threads; 20 batch threads; NUMA distribute; CPU-only full Gemma 4 draft with the native f714bf0 backend. Results and evidence `gemma26-r720-full-remote-w1-smoke-20260728` Task text generation Status PROVISIONAL Measured 2026-07-28T22:28:02Z Statistic single; 1 sample Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts forge,r720 Hardware CPU Context window 4096 Tested depth 0 Prompt tokens 20 Generated tokens 32 Prefill 20.4 tok/s Decode 7.1 tok/s Baseline g26rb (-25.2%) Output matches baseline yes Evidence results/20260728-gemma4-26b-remote-draft/smoke-speculative-full-f714-t12.log Benchmark notes. Full matching Gemma 4 draft on R720 CPU. The output was exact, but the result was decisively slower, so no five-sample sweep followed. Result notes. The remote run was 25.2% slower. Baseline and remote output SHA-256 was d6f39bca997971a5e73317d566e00114ef1c1f6362d8036253eb1718f9407a4c.
	llama.cpp/laguna	V100	TODO	128	—	—	—	—	776.5	81.0	NA	NA
`g26v` Legacy configuration for gemma26-v1001 result · 2 techniques · 0 parameters Configuration ID gemma26-v100 Created 2026-07-24T19:25:03Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations GPU layer offload The complete Gemma 4 26B-A4B target fit on the V100 with a 1 GiB margin. Weight quantization The target artifact used UD-Q4_K_XL weights. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `gemma26-v100` Task text generation Status VALID Measured 2026-07-24T19:25:03Z Statistic mean; 3 samples Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Prompt tokens 128 Generated tokens 16 Prefill 776.5 tok/s Decode 81.0 tok/s Evidence gemma-xeon/benchmark-v100-results.jsonl Benchmark notes. Full V100 offload with a 1 GiB fit margin. Result notes. pp128 and tg16 means from three llama-bench repetitions.
	llama.cpp/poolside	V100	4096	20	—	—	—	—	336.3	77.4	NA	NA
`g26vb` Legacy configuration for gemma26-v100-mtp-baseline1 result · 2 techniques · 0 parameters Configuration ID gemma26-v100-mtp-baseline Created 2026-07-24T19:31:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Flash attention The matched Gemma 4 26B-A4B V100 control used flash attention. KV-cache quantization The matched V100 control used Q8_0 K and V caches. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `gemma26-v100-mtp-baseline` Task text generation Status VALID Measured 2026-07-24T19:31:00Z Statistic median; 3 samples Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 20 Generated tokens 128 Prefill 336.3 tok/s Decode 77.4 tok/s Evidence gemma-xeon/benchmark-mtp-v100.txt Benchmark notes. Matched Poolside-build baseline with full V100 offload. Result notes. Median of three matched server requests.
	llama.cpp/poolside	CPU + V100	4096	20	—	Local CPU MTP	-32.3%	61.5%	330.3	52.4	NA	NA
`g26vw` Legacy configuration for gemma26-v100-mtp-w11 result · 4 techniques · 0 parameters Configuration ID gemma26-v100-mtp-w1 Created 2026-07-24T19:35:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Draft placement The target used the V100 and the MTP assistant used the R720 CPUs. Flash attention The Gemma 4 26B-A4B MTP benchmark used flash attention. KV-cache quantization The MTP benchmark used Q8_0 K and V caches. Speculative decoding The MTP head proposed one token per target verification step. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute r720 — draft; CPU; 0 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `gemma26-v100-mtp-w1` Task text generation Status PROVISIONAL Measured 2026-07-24T19:35:00Z Statistic median; 3 samples Model Gemma 4 26B-A4B IT Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 20 Generated tokens 128 Prefill 330.3 tok/s Decode 52.4 tok/s Baseline g26vb (-32.3%) Output matches baseline no Evidence gemma-xeon/benchmark-mtp-v100.txt Benchmark notes. CPU MTP assistant; output did not match the baseline. Result notes. Median of three requests; all three had 48/78 acceptance.

Qwen3.8

Size: 27B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(2)	4096	11	—	—	—	—	93.1	32.7	NA	62%
`q38b2` Qwen3.8 27B Q4_K_M split 1:1 across two V100s without speculation.1 result · 4 techniques · 12 parameters Configuration ID qwen38-llamab64739e-2xv100-layer-b512-c4k-f16-fa Created 2026-08-14T19:42:22Z Notes One llama.cpp server process with layer split mode and interleaved host memory. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete target was offloaded across the two V100s. NUMA memory policy The one target process used interleaved host memory. Same-host layer split One target process used a 1:1 layer split across both V100s. Uncommon parameters node:r720:numa0.free_memory_before 4358 MB MB — Recorded after service release and checksum verification. node:r720:numa1.free_memory_before 1866 MB MB — Recorded after service release and checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.cpu_governor schedutil — Recorded after GPU consumers released the host. system.gpu_service_state benchmark consumers stopped; thermal control active — All consumer services were restored after the run. system.numa_memory_policy interleave=all — The server process interleaved allocations across nodes. system.process_count 1 processes — One server process used both physical V100s. target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.fit_reserve_per_device 1536 MiB MiB — Applied to each visible V100. target.model_page_cache fully resident — All 4,176,459 file pages were resident before the run. target.model_revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 — Pinned Qwen/Qwen3.8-27B base-model revision. target.tensor_split 1,1 — Equal layer-placement weights for CUDA0 and CUDA1. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0; main GPU for the one-process 1:1 layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1; second device for the one-process 1:1 layer split. Results and evidence `qwen38-2xv100-layer-baseline-20260814` Task text generation Status VALID Measured 2026-08-14T19:42:28Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 128 Prefill 93.1 tok/s Decode 32.7 tok/s Output matches baseline yes Evidence qwen38/results/20260814T194122Z-qwen38-27b-mtp-2x-v100 Benchmark notes. Five server samples after one warmup. The fixed greedy request generated 128 tokens in every sample. Result notes. Median of five 128-token server responses. All baseline samples returned the same greedy token sequence. Strict JSON passed. The arithmetic value was correct, but its integer-only format instruction was not followed.
	llama.cpp	V100(2)	4096	512	—	—	—	—	873.0	25.2	38%	48%
`q38rep` Qwen3.8 27B Q4_K_M with one full replica on each R720 V100.1 result · 4 techniques · 16 parameters Configuration ID qwen38-llamab64739e-2xv100-replicas-b512-c4k-f16-fa Created 2026-08-14T15:57:10Z Notes Two concurrent llama.cpp processes. Each process used ten CPU threads and memory from the NUMA node local to its assigned physical GPU. Applied optimizations Flash attention Both retained throughput streams report flash_attn=true. GPU layer offload Each replica fit completely on its assigned V100. Independent model replicas One complete model process ran on each physical V100. NUMA memory policy Each process used CPU and memory binding local to its V100. Uncommon parameters accelerator:r720-v100-32gb.application_clocks automatic — No fixed application clocks were active. accelerator:r720-v100-32gb.ecc enabled accelerator:r720-v100-32gb-2.application_clocks automatic — No fixed application clocks were active. accelerator:r720-v100-32gb-2.ecc enabled backend.model_load mmap — Both llama-bench streams report use_mmap=true. node:r720:numa0.free_memory_before 2384 MB MB — Recorded after service release and model checksum verification. node:r720:numa1.free_memory_before 1499 MB MB — Recorded after service release and model checksum verification. system.aggregate_definition sum of per-replica medians — Each replica supplied five samples per phase. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.cpu_governor schedutil — Recorded after GPU consumers released the host. system.gpu_service_state benchmark consumers stopped; thermal control active — The consumer services were restored after the run. system.replica_count 2 replicas — Two independent processes ran concurrently. target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.fit_reserve 1536 MiB MiB — Applied independently to each replica. target.model_page_cache fully resident — All 4,160,011 model pages were resident before the run. target.model_revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 — Pinned Qwen/Qwen3.8-27B base-model revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 10 threads; 10 batch threads; affinity NUMA node 0 CPUs; NUMA cpunodebind=0; membind=0; llama NUMA numactl; Physical GPU 0 ran one complete, independent model replica. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 10 threads; 10 batch threads; affinity NUMA node 1 CPUs; NUMA cpunodebind=1; membind=1; llama NUMA numactl; Physical GPU 1 ran one complete, independent model replica. Results and evidence `qwen38-2xv100-replicas-20260814` Task text generation Status VALID Measured 2026-08-14T19:06:28Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 512 Generated tokens 128 Prefill 873.0 tok/s Decode 25.2 tok/s Evidence qwen38/results/20260814T190531Z-qwen38-27b-2x-v100 Benchmark notes. Each reported system rate is the sum of two per-replica medians. Each replica contributed five raw samples. Prompt throughput had high variance. GPU 1 completed its final generation sample after GPU 0, so the median is used instead of the mean. Both JSONL streams report flash_attn=true. Result notes. GPU 0 medians were 418.020 prompt and 13.8178 generation tokens/s. GPU 1 medians were 454.971 prompt and 11.3764 generation tokens/s. All three deterministic checks had identical token IDs on the two replicas. Strict JSON parsed and matched. The arithmetic value was 3919, but the response included an explanation despite the integer-only instruction. No earlier semantic baseline exists.
	llama.cpp	V100(2)	4096	11	11	—	—	—	22.3	34.1	NA	65%
	llama.cpp	V100	4096	34	34	—	—	—	59.5	33.9	NA	65%
`q383so` Qwen3.8 27B Q4_K_M on one V100 with the b10358 build and no speculation.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama030ebb558-1v100-b512-c4k-f16-fa Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b10358-single-off-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 59.5 tok/s Decode 33.9 tok/s Baseline q389so (+2.9%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10358 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 1.0% of the minimum, from 33.813 to 34.162 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.813, maximum 34.162, spread 1.0%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples. `qwen38-b10358-single-off-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 22.3 tok/s Decode 34.1 tok/s Baseline q389so (+3.4%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10358 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 0.5% of the minimum, from 33.991 to 34.175 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.991, maximum 34.175, spread 0.5%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100(2)	4096	11	11	—	—	—	23.6	33.9	NA	64%
	llama.cpp	V100	4096	34	34	—	—	—	86.0	32.5	NA	62%
`q384so1` Qwen3.8 27B Q4_K_M on the x16-linked V100 with b10488 and no speculation.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c4k-f16-fa-x16 Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA1 sits behind the x16 root port at 42:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x16 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x16 root port 40:02.0. Results and evidence `qwen38-b10488-single-off-cuda1-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 86.0 tok/s Decode 32.5 tok/s Baseline q384so (-4.1%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA1, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 3.0% of the minimum, from 32.301 to 33.263 tokens per second. Result notes. Median of five 400-token samples. Minimum 32.301, maximum 33.263, spread 3.0%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples. `qwen38-b10488-single-off-cuda1-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 23.6 tok/s Decode 33.9 tok/s Baseline q384so (+0.1%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA1, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 1.6% of the minimum, from 33.555 to 34.091 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.555, maximum 34.091, spread 1.6%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100(2)	4096	11	11	—	—	—	30.3	33.8	NA	64%
	llama.cpp	V100	4096	34	34	—	—	—	59.8	33.7	NA	64%
`q384son` Qwen3.8 27B Q4_K_M on one V100 with b10488 and CUDA graph capture disabled.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c4k-f16-fa-nograph Created 2026-08-19T01:29:18Z Notes GGML_CUDA_DISABLE_GRAPHS=1 in the unit environment. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs disabled — Graph capture disabled through the unit environment. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b10488-single-off-nograph-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 59.8 tok/s Decode 33.7 tok/s Baseline q384so (-0.5%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs disabled. Median of five samples on the code prompt; samples spanned 1.3% of the minimum, from 33.479 to 33.905 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.479, maximum 33.905, spread 1.3%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples. `qwen38-b10488-single-off-nograph-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 30.3 tok/s Decode 33.8 tok/s Baseline q384so (-0.3%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs disabled. Median of five samples on the prose prompt; samples spanned 0.9% of the minimum, from 33.640 to 33.955 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.640, maximum 33.955, spread 0.9%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100(2)	4096	11	11	—	—	—	21.6	33.9	NA	64%
	llama.cpp	V100	4096	34	34	—	—	—	87.4	33.9	NA	64%
`q384so` Qwen3.8 27B Q4_K_M on one V100 with the b10488 build and no speculation.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c4k-f16-fa Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b10488-single-off-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 87.4 tok/s Decode 33.9 tok/s Baseline q389so (+2.8%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 0.8% of the minimum, from 33.769 to 34.042 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.769, maximum 34.042, spread 0.8%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples. `qwen38-b10488-single-off-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 21.6 tok/s Decode 33.9 tok/s Baseline q389so (+2.8%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 0.5% of the minimum, from 33.807 to 33.985 tokens per second. Result notes. Median of five 400-token samples. Minimum 33.807, maximum 33.985, spread 0.5%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100(2)	4096	11	11	—	—	—	76.0	33.0	NA	63%
	llama.cpp	V100	4096	34	34	—	—	—	198.0	33.0	NA	63%
`q389so` Qwen3.8 27B Q4_K_M on one V100 with the nixpkgs b9190 build and no speculation.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llamab64739e-1v100-b512-c4k-f16-fa Created 2026-08-19T01:29:18Z Notes This build requires compute capability 8.0 before it captures a CUDA graph, so the V100 runs none. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs unavailable — The build gates graph capture at compute capability 8.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b9190-single-off-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 198.0 tok/s Decode 33.0 tok/s Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b9190 build, single placement on CUDA0, CUDA graphs unavailable. Median of five samples on the code prompt; samples spanned 1.0% of the minimum, from 32.727 to 33.060 tokens per second. Result notes. Median of five 400-token samples. Minimum 32.727, maximum 33.060, spread 1.0%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples. `qwen38-b9190-single-off-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 76.0 tok/s Decode 33.0 tok/s Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b9190 build, single placement on CUDA0, CUDA graphs unavailable. Median of five samples on the prose prompt; samples spanned 0.8% of the minimum, from 32.804 to 33.076 tokens per second. Result notes. Median of five 400-token samples. Minimum 32.804, maximum 33.076, spread 0.8%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100(2)	4096	11	11	—	—	—	27.2	33.6	NA	64%
	llama.cpp	V100(2)	4096	11	11	—	—	—	20.7	33.2	NA	63%
	llama.cpp	V100(2)	4096	11	11	—	—	—	24.9	34.2	NA	65%
	llama.cpp	V100(2)	4096	11	11	—	—	—	33.9	33.6	NA	64%
	llama.cpp	V100(2)	4096	20	20	—	—	—	49.9	33.9	NA	65%
	llama.cpp	V100(2)	4096	24	24	—	—	—	46.8	33.8	NA	64%
	llama.cpp	V100	4096	34	34	—	—	—	68.1	33.0	NA	63%
	llama.cpp	V100	4096	34	34	—	—	—	84.2	33.4	NA	63%
	llama.cpp	V100	4096	34	34	—	—	—	65.9	34.1	NA	65%
	llama.cpp	V100(2)	4096	34	34	—	—	—	94.0	33.5	NA	64%
`q38n0` Qwen3.8 27B Q4_K_M on one V100 without speculation, b10358.10 results · 1 technique · 2 parameters Configuration ID qwen38-llamab10358-v100-single-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-confirm-1gpu-off-arithmetic-20260817` Task text generation Status VALID Measured 2026-08-17T07:30:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 20 Prompt tokens 20 Generated tokens 38 Prefill 49.9 tok/s Decode 33.9 tok/s Time to first token 401.1 ms Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the arithmetic task; samples spanned 2.2% of the minimum. Result notes. Median decode of 5 samples between 33.929 and 34.668 tok/s. Every sample matched the matched non-speculative arm. `qwen38-confirm-1gpu-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T07:30:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 68.1 tok/s Decode 33.0 tok/s Time to first token 499.4 ms Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the code prompt; samples spanned 1.8% of the minimum. Result notes. Median decode of 5 samples between 32.896 and 33.485 tok/s. Every sample matched the matched non-speculative arm. `qwen38-confirm-1gpu-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T07:30:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 27.2 tok/s Decode 33.6 tok/s Time to first token 404.4 ms Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the prose prompt; samples spanned 2.2% of the minimum. Result notes. Median decode of 5 samples between 33.482 and 34.212 tok/s. Every sample matched the matched non-speculative arm. `qwen38-confirm-1gpu-off-strict-json-20260817` Task text generation Status VALID Measured 2026-08-17T07:30:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 24 Prompt tokens 24 Generated tokens 22 Prefill 46.8 tok/s Decode 33.8 tok/s Time to first token 512.7 ms Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the strict JSON task; samples spanned 5.3% of the minimum. Result notes. Median decode of 5 samples between 33.177 and 34.944 tok/s. Every sample matched the matched non-speculative arm. `qwen38-len1024-1gpu-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:59:14Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 1024 Prefill 84.2 tok/s Decode 33.4 tok/s Time to first token 403.7 ms Output matches baseline yes Evidence qwen38/results/20260817T065907Z-qwen38-27b-mtp-length-1024 Benchmark notes. Generated-length series at 1,024 tokens on the b10358 build. Median of 3 samples on the code prompt; samples spanned 0.2% of the minimum. Result notes. Median decode of 3 samples between 33.360 and 33.424 tok/s. Every sample matched the matched non-speculative arm. `qwen38-len1024-1gpu-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:59:14Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 1024 Prefill 20.7 tok/s Decode 33.2 tok/s Time to first token 530.2 ms Output matches baseline yes Evidence qwen38/results/20260817T065907Z-qwen38-27b-mtp-length-1024 Benchmark notes. Generated-length series at 1,024 tokens on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 4.9% of the minimum. Result notes. Median decode of 3 samples between 31.765 and 33.332 tok/s. Every sample matched the matched non-speculative arm. `qwen38-len128-1gpu-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:56:39Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 128 Prefill 65.9 tok/s Decode 34.1 tok/s Time to first token 515.8 ms Output matches baseline yes Evidence qwen38/results/20260817T065631Z-qwen38-27b-mtp-length-128 Benchmark notes. Generated-length series at 128 tokens on the b10358 build. Median of 3 samples on the code prompt; samples spanned 3.5% of the minimum. Result notes. Median decode of 3 samples between 33.251 and 34.415 tok/s. Every sample matched the matched non-speculative arm. `qwen38-len128-1gpu-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:56:39Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 128 Prefill 24.9 tok/s Decode 34.2 tok/s Time to first token 441.9 ms Output matches baseline yes Evidence qwen38/results/20260817T065631Z-qwen38-27b-mtp-length-128 Benchmark notes. Generated-length series at 128 tokens on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 0.2% of the minimum. Result notes. Median decode of 3 samples between 34.154 and 34.217 tok/s. Every sample matched the matched non-speculative arm. `qwen38-width-1gpu-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:40:11Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 94.0 tok/s Decode 33.5 tok/s Time to first token 361.7 ms Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 1.1% of the minimum. Result notes. Median decode of 3 samples between 33.514 and 33.899 tok/s. Every sample matched the matched non-speculative arm. `qwen38-width-1gpu-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:40:11Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 33.9 tok/s Decode 33.6 tok/s Time to first token 324.8 ms Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 1.6% of the minimum. Result notes. Median decode of 3 samples between 33.582 and 34.117 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	4096	11	11	—	—	—	—	33.2	NA	63%
	llama.cpp	V100	4096	34	34	—	—	—	—	33.3	NA	63%
`q38d2o` Qwen3.8 27B Q4_K_M on one V100 at a 4,096-token context with batch and microbatch 512, Flash Attention, F16 KV, and no speculative decoding.2 results · 0 techniques · 2 parameters Configuration ID qwen38-llama5ecbe1ac1-1v100-b512-c4k-f16-fa-off Created 2026-08-20T06:39:50Z Notes Served by the DFlash2 llama.cpp build so that every arm of this experiment shares one runtime. Applied optimizations No applied technique is cataloged. Uncommon parameters system.device_memory_used 16202 MiB MiB — Reported by nvidia-smi after the model loaded. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; One V100 with --split-mode none. Results and evidence `qwen38-dflash2-off-code-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Decode 33.3 tok/s Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The code prompt at 400 generated tokens. Median of 5 samples; the samples spanned 2.1% of the minimum. Result notes. Decode median of five samples on the code prompt. `qwen38-dflash2-off-prose-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Decode 33.2 tok/s Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The prose prompt at 400 generated tokens. Median of 5 samples; the samples spanned 7.5% of the minimum. Result notes. Decode median of five samples on the prose prompt.
	llama.cpp	V100(2)	4096	11	11	—	—	—	92.6	32.7	NA	62%
	llama.cpp	V100(2)	4096	11	11	—	—	—	93.3	33.0	NA	63%
	llama.cpp	V100(2)	4096	34	34	—	—	—	239.9	32.9	NA	63%
	llama.cpp	V100(2)	4096	34	34	—	—	—	241.4	33.1	NA	63%
`q38l0` Qwen3.8 27B Q4_K_M split 1:1 across two V100s, b9190.4 results · 2 techniques · 2 parameters Configuration ID qwen38-llamab9190-2xv100-layer-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Same-host layer split One process with a 1:1 layer split across both V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-repro-2gpu-layer-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T07:37:46Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 128 Prefill 239.9 tok/s Decode 32.9 tok/s Time to first token 141.7 ms Output matches baseline yes Evidence qwen38/results/20260817T073738Z-qwen38-27b-mtp-repro-b9190-128 Benchmark notes. Reproduction of the invalid 2026-08-14 arm under its own build, placement, and generated length. Median of 5 samples on the code prompt; samples spanned 1.0% of the minimum. Result notes. Median decode of 5 samples between 32.886 and 33.221 tok/s. Every sample matched the matched non-speculative arm. `qwen38-repro-2gpu-layer-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T07:37:46Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 128 Prefill 92.6 tok/s Decode 32.7 tok/s Time to first token 118.8 ms Output matches baseline yes Evidence qwen38/results/20260817T073738Z-qwen38-27b-mtp-repro-b9190-128 Benchmark notes. Reproduction of the invalid 2026-08-14 arm under its own build, placement, and generated length. Median of 5 samples on the prose prompt; samples spanned 2.1% of the minimum. Result notes. Median decode of 5 samples between 32.220 and 32.904 tok/s. Every sample matched the matched non-speculative arm. `qwen38-screen-2gpu-layer-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:30:01Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 241.4 tok/s Decode 33.1 tok/s Time to first token 140.9 ms Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the code prompt; samples spanned 0.2% of the minimum. Result notes. Median decode of 3 samples between 33.031 and 33.090 tok/s. Every sample matched the matched non-speculative arm. `qwen38-screen-2gpu-layer-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:30:01Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 93.3 tok/s Decode 33.0 tok/s Time to first token 117.9 ms Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the prose prompt; samples spanned 0.1% of the minimum. Result notes. Median decode of 3 samples between 32.948 and 32.996 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	4096	11	11	—	—	—	75.9	33.1	NA	63%
	llama.cpp	V100	4096	34	34	—	—	—	202.1	32.9	NA	62%
`q38s0` Qwen3.8 27B Q4_K_M on one V100 without speculation, b9190.2 results · 1 technique · 2 parameters Configuration ID qwen38-llamab9190-v100-single-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-screen-1gpu-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:26:33Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 202.1 tok/s Decode 32.9 tok/s Time to first token 168.2 ms Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the code prompt; samples spanned 0.8% of the minimum. Result notes. Median decode of 3 samples between 32.610 and 32.871 tok/s. Every sample matched the matched non-speculative arm. `qwen38-screen-1gpu-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:26:33Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 75.9 tok/s Decode 33.1 tok/s Time to first token 144.8 ms Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the prose prompt; samples spanned 0.8% of the minimum. Result notes. Median decode of 3 samples between 32.852 and 33.101 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	4096	11	11	—	—	—	24.7	31.9	NA	30%
	llama.cpp	V100(2)	4096	34	34	—	—	—	75.5	30.5	NA	29%
`q38t0` Qwen3.8 27B Q4_K_M tensor split across two V100s, b9190.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab9190-2xv100-tensor-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. GPU link topology Every all-reduce crossed the SYS path between the cards. Tensor parallelism One process with a tensor split across both V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-screen-2gpu-tensor-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:33:29Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 75.5 tok/s Decode 30.5 tok/s Time to first token 450.2 ms Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the code prompt; samples spanned 91.2% of the minimum. Result notes. Median decode of 3 samples between 17.842 and 34.117 tok/s. Every sample matched the matched non-speculative arm. `qwen38-screen-2gpu-tensor-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:33:29Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 24.7 tok/s Decode 31.9 tok/s Time to first token 444.6 ms Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the prose prompt; samples spanned 82.3% of the minimum. Result notes. Median decode of 3 samples between 18.568 and 33.856 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	4096	11	11	—	—	—	28.1	34.2	NA	65%
	llama.cpp	V100(2)	4096	34	34	—	—	—	87.2	34.2	NA	65%
`q38d0` Qwen3.8 27B Q4_K_M split 1:1 across two V100s, b10358.2 results · 2 techniques · 2 parameters Configuration ID qwen38-llamab10358-2xv100-layer-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Same-host layer split One process with a 1:1 layer split across both V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-width-2gpu-layer-off-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:51:27Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 87.2 tok/s Decode 34.2 tok/s Time to first token 389.9 ms Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 0.7% of the minimum. Result notes. Median decode of 3 samples between 34.068 and 34.292 tok/s. Every sample matched the matched non-speculative arm. `qwen38-width-2gpu-layer-off-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:51:27Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 28.1 tok/s Decode 34.2 tok/s Time to first token 392.1 ms Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 0.3% of the minimum. Result notes. Median decode of 3 samples between 34.198 and 34.287 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	40960	30906	30906	—	—	—	589.0	27.9	26%	60%
`q38p0` Qwen3.8 27B Q4_K_M on one V100 at a 40,960-token context, b10358.1 result · 1 technique · 2 parameters Configuration ID qwen38-llamab10358-v100-single-b512-c40k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-depth-1gpu-off-depth-20260817` Task text generation Status VALID Measured 2026-08-17T07:12:57Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 589.0 tok/s Decode 27.9 tok/s Time to first token 52475.2 ms Output matches baseline yes Evidence qwen38/results/20260817T071250Z-qwen38-27b-mtp-depth-32k Benchmark notes. Deep-context pair. The prompt is 100,000 bytes of this repository's tracked Markdown in a fixed order. Median of 3 samples on the deep-context prompt; samples spanned 2.2% of the minimum. Result notes. Median decode of 3 samples between 27.465 and 28.082 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	40960	30906	30906	—	—	—	603.2	28.0	26%	61%
`q38q0` Qwen3.8 27B Q4_K_M split 1:1 at a 40,960-token context, b10358.1 result · 2 techniques · 2 parameters Configuration ID qwen38-llamab10358-2xv100-layer-b512-c40k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Same-host layer split One process with a 1:1 layer split across both V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-depth-2gpu-layer-off-depth-20260817` Task text generation Status VALID Measured 2026-08-17T07:21:26Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 603.2 tok/s Decode 28.0 tok/s Time to first token 51233.0 ms Output matches baseline yes Evidence qwen38/results/20260817T071250Z-qwen38-27b-mtp-depth-32k Benchmark notes. Deep-context pair. The prompt is 100,000 bytes of this repository's tracked Markdown in a fixed order. Median of 3 samples on the deep-context prompt; samples spanned 2.6% of the minimum. Result notes. Median decode of 3 samples between 27.710 and 28.431 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	40960	30906	30906	—	—	—	687.1	28.0	30%	61%
`q38ub1` Qwen3.8 27B Q4_K_M on one V100 with microbatch 1024 and logical batch 2048.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b1024-c40k-f16-fa-depth31k Created 2026-08-19T02:05:05Z Notes Cold prefill of a 30,906-token prompt. The server reported 18704 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 18704 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth-ub1024-20260819` Task text generation Status VALID Measured 2026-08-19T02:05:05Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 687.1 tok/s Decode 28.0 tok/s Baseline q38ub5 (+15.4%) Output matches baseline yes Evidence qwen38/results/20260819T020505Z-qwen38-27b-depth-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 1024. Samples spanned 0.3% of the minimum, from 685.6 to 687.7 tokens per second. That is 1.154 times the 512 arm. Result notes. Cold prefill at a 30,906-token prompt with the host prompt cache disabled. Minimum 685.6, maximum 687.7. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	40960	30906	30906	—	—	—	743.7	28.1	33%	61%
`q38ub2` Qwen3.8 27B Q4_K_M on one V100 with microbatch 2048 and logical batch 2048.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b2048-c40k-f16-fa-depth31k Created 2026-08-19T02:05:05Z Notes Cold prefill of a 30,906-token prompt. The server reported 19028 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 19028 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth-ub2048-20260819` Task text generation Status VALID Measured 2026-08-19T02:05:05Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 743.7 tok/s Decode 28.1 tok/s Baseline q38ub5 (+24.9%) Output matches baseline yes Evidence qwen38/results/20260819T020505Z-qwen38-27b-depth-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 2048. Samples spanned 0.2% of the minimum, from 743.3 to 744.7 tokens per second. That is 1.249 times the 512 arm. Result notes. Cold prefill at a 30,906-token prompt with the host prompt cache disabled. Minimum 743.3, maximum 744.7. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	40960	30906	30906	—	—	—	778.1	28.1	34%	61%
`q38ub4` Qwen3.8 27B Q4_K_M on one V100 with microbatch 4096 and logical batch 4096.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b4096-c40k-f16-fa-depth31k Created 2026-08-19T02:05:05Z Notes Cold prefill of a 30,906-token prompt. The server reported 19676 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 19676 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth-ub4096-20260819` Task text generation Status VALID Measured 2026-08-19T02:05:05Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 778.1 tok/s Decode 28.1 tok/s Baseline q38ub5 (+30.7%) Output matches baseline yes Evidence qwen38/results/20260819T020505Z-qwen38-27b-depth-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 4096. Samples spanned 0.6% of the minimum, from 775.8 to 780.8 tokens per second. That is 1.307 times the 512 arm. Result notes. Cold prefill at a 30,906-token prompt with the host prompt cache disabled. Minimum 775.8, maximum 780.8. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	40960	30906	30906	—	—	—	595.5	28.1	26%	61%
`q38ub5` Qwen3.8 27B Q4_K_M on one V100 with microbatch 512 and logical batch 2048.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c40k-f16-fa-depth31k Created 2026-08-19T02:05:05Z Notes Cold prefill of a 30,906-token prompt. The server reported 18542 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 18542 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth-ub512-20260819` Task text generation Status VALID Measured 2026-08-19T02:05:05Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 595.5 tok/s Decode 28.1 tok/s Output matches baseline yes Evidence qwen38/results/20260819T020505Z-qwen38-27b-depth-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 512. Samples spanned 0.2% of the minimum, from 594.8 to 596.0 tokens per second. That is 1.000 times the 512 arm. Result notes. Cold prefill at a 30,906-token prompt with the host prompt cache disabled. Minimum 594.8, maximum 596.0. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	32768	30906	30906	—	—	—	591.1	27.8	26%	60%
`q38v0` Qwen3.8 27B Q4_K_M on one V100 at the 32,768-token router context, b10358.1 result · 1 technique · 2 parameters Configuration ID qwen38-llamab10358-v100-single-b512-c32k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-serving-1gpu-off-depth-20260817` Task text generation Status VALID Measured 2026-08-17T07:41:53Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 591.1 tok/s Decode 27.8 tok/s Time to first token 52284.3 ms Output matches baseline yes Evidence qwen38/results/20260817T074146Z-qwen38-27b-mtp-serving-32k Benchmark notes. Serving-profile check at the router context and depth. Median of 3 samples on the deep-context prompt; samples spanned 1.7% of the minimum. Result notes. Median decode of 3 samples between 27.625 and 28.097 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	4096	11	—	Local GPU MTP	+3.6%	68.9%	68.6	33.9	NA	NA
`q38m2` Qwen3.8 27B Q4_K_M split 1:1 across two V100s with native MTP.1 result · 5 techniques · 17 parameters Configuration ID qwen38-llamab64739e-2xv100-layer-b512-c4k-f16-fa-mtp2-p0 Created 2026-08-14T19:43:11Z Notes Matched target placement with draft-mtp width two and minimum draft probability zero. Applied optimizations Flash attention Flash Attention was enabled with F16 target and draft cache types. GPU layer offload The complete target and native MTP work used the V100 pair. NUMA memory policy The one target process used interleaved host memory. Same-host layer split The matched target retained the same 1:1 two-V100 layer split. Speculative decoding The built-in MTP head drafted two tokens per draft call. Uncommon parameters draft.implementation draft-mtp — llama.cpp created the MTP draft context from the target artifact. draft.kv_cache_k F16 draft.kv_cache_v F16 draft.minimum_probability 0 draft.width 2 tokens node:r720:numa0.free_memory_before 4358 MB MB — Recorded after service release and checksum verification. node:r720:numa1.free_memory_before 1866 MB MB — Recorded after service release and checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.cpu_governor schedutil — Recorded after GPU consumers released the host. system.gpu_service_state benchmark consumers stopped; thermal control active — All consumer services were restored after the run. system.numa_memory_policy interleave=all — The server process interleaved allocations across nodes. system.process_count 1 processes — One server process used both physical V100s. target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.fit_reserve_per_device 1536 MiB MiB — Applied to each visible V100. target.model_page_cache fully resident — All 4,176,459 file pages were resident before the run. target.model_revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 — Pinned Qwen/Qwen3.8-27B base-model revision. target.tensor_split 1,1 — Equal layer-placement weights for CUDA0 and CUDA1. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0; main GPU for the target and native MTP context. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1; second device for the target and native MTP context. Results and evidence `qwen38-2xv100-layer-mtp2-20260814` Task text generation Status INVALID Measured 2026-08-14T19:43:11Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 128 Prefill 68.6 tok/s Decode 33.9 tok/s Baseline q38b2 (+3.6%) Output matches baseline yes Evidence qwen38/results/20260814T194122Z-qwen38-27b-mtp-2x-v100 Benchmark notes. Native draft-mtp head with width two and minimum draft probability zero. Five server samples followed one warmup. This arm is invalid: its five samples spread 23% and its median understated the configuration. The 2026-08-17 reproduction of the same build, placement, and generated length measures 41.560 tok/s against a 32.677 tok/s baseline, a 1.27x gain, with the speculative samples inside 0.7% and the baseline samples inside 2.1%. Result notes. MTP improved median decode by 3.64%. Every retained sample drafted 106 tokens and accepted 73. All five token sequences matched the baseline exactly. Strict JSON passed. The arithmetic value was correct, but its integer-only format instruction was not followed. The speedup of this row is not a measurement of the configuration. See the 2026-08-17 reproduction.
	llama.cpp	V100	131072	TODO	—	Local GPU MTP	—	—	—	—	NA	NA
	llama.cpp	V100	131072	111781	111781	Local GPU MTP	+2.5%	—	425.2	25.4	19%	NA
	llama.cpp	V100	131072	112941	112941	Local GPU MTP	+32.4%	—	249.4	32.9	11%	NA
	llama.cpp	V100	131072	116943	116943	Local GPU MTP	+11.8%	—	420.2	27.8	18%	NA
	llama.cpp	V100	131072	118103	118103	Local GPU MTP	-2.4%	—	243.1	24.2	11%	NA
`q38cr64` Qwen3.8 27B Q4_K_M on one V100 at a 131,072-token context with microbatch 4096, the MTP head at width two, and --cache-ram 65536 MiB.5 results · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b4096-c128k-f16-fa-mtp2-cram65536 Created 2026-08-19T02:31:51Z Notes Large enough for both session prefixes. The server reported a prompt state of 8,739 MiB for one session of about 112,000 tokens. Applied optimizations No applied technique is cataloged. Uncommon parameters system.host_prompt_cache enabled MiB — --cache-ram value for the arm. system.model_cache_state warm — The artifact was resident in the host page cache. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. system.retained_prefixes 7 prefixes — Whole 112,000-token prefixes the budget can hold at the measured 8,739 MiB each. system.server_slots 1 slots — --parallel 1; the router serves one slot. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); Physical GPU 0 on the x16 root port since 2026-08-21. The command line was not retained; the harness defaults match the 20260819 baseline nodes. Results and evidence `qwen38-cache-x16-cram65536-20260821` Task text generation Status INVALID Measured 2026-08-21T17:22:12Z Statistic single; 1 sample Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Evidence qwen38/results/20260821T172048Z-qwen38-27b-prompt-cache-x16 Benchmark notes. Aborted before the first sample: no response, server log, or timing was retained, and host-restored.txt follows host-before.txt by one second. The planned sequence and settings match the retained qwen38-cache-cram65536-* baseline; sample_count 1 is the schema minimum and zero samples were retained. started_at is from run.env. The cause of the abort was not retained. Cataloged 2026-08-29 from the recovered run directory. `qwen38-cache-cram65536-a1-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 425.2 tok/s Decode 25.4 tok/s Time to first token 262906.7 ms Baseline q38cr0 (-1.0%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session A first request; a cold prefill in every arm. --cache-ram 65536 MiB. The timed request was a prefix-cache miss: 0 of 111781 prompt tokens were already processed. Median of 5 samples; time to first token spanned 1.4% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 260046 ms, maximum 263612 ms. 0 of 111781 prompt tokens came from cache. `qwen38-cache-cram65536-a2-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 112941 Prompt tokens 112941 Generated tokens 32 Prefill 249.4 tok/s Decode 32.9 tok/s Time to first token 21076.1 ms Baseline q38cr0 (+1152.2%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session A returns with one appended turn. --cache-ram 65536 MiB. The timed request was a prefix-cache hit: 107685 of 112941 prompt tokens were already processed, leaving 5256 to process, because the reusable prefix is truncated to a whole microbatch. Median of 5 samples; time to first token spanned 0.4% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 21042 ms, maximum 21125 ms. 107685 of 5256 prompt tokens came from cache. `qwen38-cache-cram65536-b1-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 116943 Prompt tokens 116943 Generated tokens 32 Prefill 420.2 tok/s Decode 27.8 tok/s Time to first token 278285.7 ms Baseline q38cr0 (+0.1%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session B first request; takes the slot from session A. --cache-ram 65536 MiB. The timed request was a prefix-cache miss: 0 of 116943 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.2% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 278032 ms, maximum 278667 ms. 0 of 116943 prompt tokens came from cache. `qwen38-cache-cram65536-b2-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 118103 Prompt tokens 118103 Generated tokens 32 Prefill 243.1 tok/s Decode 24.2 tok/s Time to first token 21622.9 ms Baseline q38cr0 (+1205.4%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session B returns with one appended turn. --cache-ram 65536 MiB. The timed request was a prefix-cache hit: 112847 of 118103 prompt tokens were already processed, leaving 5256 to process, because the reusable prefix is truncated to a whole microbatch. Median of 5 samples; time to first token spanned 1.1% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 21591 ms, maximum 21824 ms. 112847 of 5256 prompt tokens came from cache.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+42.7%	74.7%	26.8	48.7	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+43.3%	74.7%	54.2	48.9	NA	NA
`q383sm` Qwen3.8 27B Q4_K_M on one V100 with b10358 and the native MTP head at width two.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama030ebb558-1v100-b512-c4k-f16-fa-mtp2-p0 Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b10358-single-mtp2-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 54.2 tok/s Decode 48.9 tok/s Baseline q383so (+44.0%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10358 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 1.1% of the minimum, from 48.670 to 49.214 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.670, maximum 49.214, spread 1.1%. The token sequence first differs from that reference at generated token 70. This is the deterministic MTP divergence already recorded for the code prompt. `qwen38-b10358-single-mtp2-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 26.8 tok/s Decode 48.7 tok/s Baseline q383so (+42.7%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10358 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 1.0% of the minimum, from 48.409 to 48.910 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.409, maximum 48.910, spread 1.0%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+43.0%	74.7%	20.1	48.4	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+42.6%	74.7%	54.8	48.3	NA	NA
`q384lm` Qwen3.8 27B Q4_K_M split 1:1 by layer across two V100s with b10488 and the MTP head.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-2v100layer-b512-c4k-f16-fa-mtp2-p0 Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 and x16 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port; main GPU. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; NUMA interleave=all; Physical GPU 1 on the x16 root port. Results and evidence `qwen38-b10488-dual-layer-mtp2-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 54.8 tok/s Decode 48.3 tok/s Baseline q384so (+42.6%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, layer placement on CUDA0,CUDA1, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 2.5% of the minimum, from 47.516 to 48.714 tokens per second. Result notes. Median of five 400-token samples. Minimum 47.516, maximum 48.714, spread 2.5%. The token sequence first differs from that reference at generated token 70. This is the deterministic MTP divergence already recorded for the code prompt. `qwen38-b10488-dual-layer-mtp2-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 20.1 tok/s Decode 48.4 tok/s Baseline q384so (+43.0%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, layer placement on CUDA0,CUDA1, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 1.5% of the minimum, from 48.076 to 48.816 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.076, maximum 48.816, spread 1.5%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+58.4%	73.1%	18.0	53.5	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+34.3%	73.4%	50.8	45.4	NA	NA
`q384tmn` Qwen3.8 27B Q4_K_M split by tensor across two V100s with b10488, MTP, and no graphs.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-2v100tensor-b512-c4k-f16-fa-mtp2-p0-nograph Created 2026-08-19T01:29:18Z Notes GGML_CUDA_DISABLE_GRAPHS=1 in the unit environment. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs disabled — Graph capture disabled through the unit environment. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 and x16 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port; main GPU. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; NUMA interleave=all; Physical GPU 1 on the x16 root port. Results and evidence `qwen38-b10488-dual-tensor-mtp2-nograph-code-20260819` Task text generation Status PROVISIONAL Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 50.8 tok/s Decode 45.4 tok/s Baseline q384son (+34.6%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, tensor placement on CUDA0,CUDA1, CUDA graphs disabled. Median of five samples on the code prompt; samples spanned 64.5% of the minimum, from 30.714 to 50.512 tokens per second. The spread exceeds the effect under test, so this median carries no claim. Result notes. Median of five 400-token samples. Minimum 30.714, maximum 50.512, spread 64.5%. The token sequence first differs from that reference at generated token 230. Tensor split changes the reduction order, and the divergence is deterministic across samples. `qwen38-b10488-dual-tensor-mtp2-nograph-prose-20260819` Task text generation Status PROVISIONAL Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 18.0 tok/s Decode 53.5 tok/s Baseline q384son (+58.4%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, tensor placement on CUDA0,CUDA1, CUDA graphs disabled. Median of five samples on the prose prompt; samples spanned 73.1% of the minimum, from 32.253 to 55.823 tokens per second. The spread exceeds the effect under test, so this median carries no claim. Result notes. Median of five 400-token samples. Minimum 32.253, maximum 55.823, spread 73.1%. The token sequence first differs from that reference at generated token 133. Tensor split changes the reduction order, and the divergence is deterministic across samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+44.0%	73.1%	22.9	48.8	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+28.7%	73.4%	62.9	43.6	NA	NA
`q384tm` Qwen3.8 27B Q4_K_M split 1:1 by tensor across two V100s with b10488 and the MTP head.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-2v100tensor-b512-c4k-f16-fa-mtp2-p0 Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 and x16 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port; main GPU. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; NUMA interleave=all; Physical GPU 1 on the x16 root port. Results and evidence `qwen38-b10488-dual-tensor-mtp2-code-20260819` Task text generation Status PROVISIONAL Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 62.9 tok/s Decode 43.6 tok/s Baseline q384so (+28.7%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, tensor placement on CUDA0,CUDA1, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 82.2% of the minimum, from 28.330 to 51.617 tokens per second. The spread exceeds the effect under test, so this median carries no claim. Result notes. Median of five 400-token samples. Minimum 28.330, maximum 51.617, spread 82.2%. The token sequence first differs from that reference at generated token 230. Tensor split changes the reduction order, and the divergence is deterministic across samples. `qwen38-b10488-dual-tensor-mtp2-prose-20260819` Task text generation Status PROVISIONAL Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 22.9 tok/s Decode 48.8 tok/s Baseline q384so (+44.0%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, tensor placement on CUDA0,CUDA1, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 55.2% of the minimum, from 34.562 to 53.645 tokens per second. The spread exceeds the effect under test, so this median carries no claim. Result notes. Median of five 400-token samples. Minimum 34.562, maximum 53.645, spread 55.2%. The token sequence first differs from that reference at generated token 133. Tensor split changes the reduction order, and the divergence is deterministic across samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+43.7%	74.7%	24.1	48.7	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+40.2%	74.7%	74.7	47.5	NA	NA
`q384sm1` Qwen3.8 27B Q4_K_M on the x16-linked V100 with b10488 and the MTP head at width two.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c4k-f16-fa-x16-mtp2-p0 Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA1 sits behind the x16 root port at 42:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x16 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x16 root port 40:02.0. Results and evidence `qwen38-b10488-single-mtp2-cuda1-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 74.7 tok/s Decode 47.5 tok/s Baseline q384so1 (+46.3%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA1, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 1.3% of the minimum, from 47.292 to 47.907 tokens per second. Result notes. Median of five 400-token samples. Minimum 47.292, maximum 47.907, spread 1.3%. The token sequence first differs from that reference at generated token 70. This is the deterministic MTP divergence already recorded for the code prompt. `qwen38-b10488-single-mtp2-cuda1-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 24.1 tok/s Decode 48.7 tok/s Baseline q384so1 (+43.7%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA1, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 1.8% of the minimum, from 48.218 to 49.064 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.218, maximum 49.064, spread 1.8%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+44.0%	74.7%	25.8	48.7	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+43.7%	74.7%	74.8	48.6	NA	NA
`q384smn` Qwen3.8 27B Q4_K_M with b10488, the MTP head at width two, and CUDA graphs disabled.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c4k-f16-fa-mtp2-p0-nograph Created 2026-08-19T01:29:18Z Notes GGML_CUDA_DISABLE_GRAPHS=1 in the unit environment. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs disabled — Graph capture disabled through the unit environment. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b10488-single-mtp2-nograph-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 74.8 tok/s Decode 48.6 tok/s Baseline q384son (+44.1%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs disabled. Median of five samples on the code prompt; samples spanned 0.4% of the minimum, from 48.500 to 48.716 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.500, maximum 48.716, spread 0.4%. The token sequence first differs from that reference at generated token 70. This is the deterministic MTP divergence already recorded for the code prompt. `qwen38-b10488-single-mtp2-nograph-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 25.8 tok/s Decode 48.7 tok/s Baseline q384son (+44.0%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs disabled. Median of five samples on the prose prompt; samples spanned 1.1% of the minimum, from 48.213 to 48.751 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.213, maximum 48.751, spread 1.1%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+43.1%	74.7%	27.1	48.5	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+43.6%	74.7%	77.2	48.7	NA	NA
`q384sm` Qwen3.8 27B Q4_K_M on one V100 with b10488 and the native MTP head at width two.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c4k-f16-fa-mtp2-p0 Created 2026-08-19T01:29:18Z Notes The build captures CUDA graphs on Volta. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — The build gates graph capture at compute capability 7.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b10488-single-mtp2-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 77.2 tok/s Decode 48.7 tok/s Baseline q384so (+43.6%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the code prompt; samples spanned 1.0% of the minimum, from 48.230 to 48.720 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.230, maximum 48.720, spread 1.0%. The token sequence first differs from that reference at generated token 70. This is the deterministic MTP divergence already recorded for the code prompt. `qwen38-b10488-single-mtp2-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 27.1 tok/s Decode 48.5 tok/s Baseline q384so (+43.1%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b10488 build, single placement on CUDA0, CUDA graphs enabled. Median of five samples on the prose prompt; samples spanned 1.4% of the minimum, from 48.086 to 48.760 tokens per second. Result notes. Median of five 400-token samples. Minimum 48.086, maximum 48.760, spread 1.4%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+30.3%	74.7%	68.5	43.0	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+29.7%	74.7%	171.2	42.7	NA	NA
`q389sm` Qwen3.8 27B Q4_K_M on one V100 with b9190 and the native MTP head at width two.2 results · 0 techniques · 6 parameters Configuration ID qwen38-llamab64739e-1v100-b512-c4k-f16-fa-mtp2-p0 Created 2026-08-19T01:29:18Z Notes This build requires compute capability 8.0 before it captures a CUDA graph, so the V100 runs none. CUDA0 sits behind the x8 root port at 05:00.0. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs unavailable — The build gates graph capture at compute capability 8.0. system.cpu_frequency_governor schedutil system.host_prompt_cache disabled MiB — --cache-ram 0; every request processed its own prompt. system.model_cache_state warm — The artifact was resident in the host page cache. system.pcie_link_width x8 — Negotiated width of the root port behind the serving device. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; x8 root port 00:03.0. Results and evidence `qwen38-b9190-single-mtp2-code-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 171.2 tok/s Decode 42.7 tok/s Baseline q389so (+29.7%) Output matches baseline no Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b9190 build, single placement on CUDA0, CUDA graphs unavailable. Median of five samples on the code prompt; samples spanned 1.8% of the minimum, from 42.364 to 43.146 tokens per second. Result notes. Median of five 400-token samples. Minimum 42.364, maximum 43.146, spread 1.8%. The token sequence first differs from that reference at generated token 70. This is the deterministic MTP divergence already recorded for the code prompt. `qwen38-b9190-single-mtp2-prose-20260819` Task text generation Status VALID Measured 2026-08-19T01:29:18Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 68.5 tok/s Decode 43.0 tok/s Baseline q389so (+30.3%) Output matches baseline yes Evidence qwen38/results/20260819T012918Z-qwen38-27b-build-graph-device-split Benchmark notes. b9190 build, single placement on CUDA0, CUDA graphs unavailable. Median of five samples on the prose prompt; samples spanned 0.6% of the minimum, from 42.898 to 43.176 tokens per second. Result notes. Median of five 400-token samples. Minimum 42.898, maximum 43.176, spread 0.6%. The token sequence matched the b10488 single-card non-speculative reference exactly in all five samples.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+42.6%	74.7%	20.2	47.9	NA	NA
	llama.cpp	V100	4096	11	11	Local GPU MTP	+48.0%	81.3%	24.7	49.7	NA	NA
	llama.cpp	V100	4096	11	11	Local GPU MTP	+38.7%	68.9%	21.1	46.6	NA	NA
	llama.cpp	V100	4096	11	11	Local GPU MTP	+43.1%	74.7%	23.3	48.1	NA	NA
	llama.cpp	V100	4096	20	20	Local GPU MTP	+63.7%	100.0%	36.9	55.0	NA	NA
	llama.cpp	V100	4096	24	24	Local GPU MTP	+54.0%	81.3%	49.1	51.8	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+43.1%	74.7%	69.7	48.1	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+50.9%	82.7%	73.5	50.7	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+44.8%	78.6%	69.2	48.7	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+41.9%	74.7%	59.2	47.7	NA	NA
`q38n2` Qwen3.8 27B Q4_K_M on one V100 with native MTP width two, b10358.10 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp2-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-confirm-1gpu-mtp2-arithmetic-20260817` Task text generation Status VALID Measured 2026-08-17T07:34:04Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 20 Prompt tokens 20 Generated tokens 38 Prefill 36.9 tok/s Decode 55.0 tok/s Time to first token 542.1 ms Baseline q38n0 (+62.0%) Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the arithmetic task; samples spanned 11.7% of the minimum. The head drafted 130 tokens and the target accepted 130. Result notes. Median decode of 5 samples between 50.573 and 56.483 tok/s. Every sample matched the matched non-speculative arm. `qwen38-confirm-1gpu-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T07:34:04Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 69.7 tok/s Decode 48.1 tok/s Time to first token 488.0 ms Baseline q38n0 (+45.5%) Output matches baseline no Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the code prompt; samples spanned 1.5% of the minimum. The head drafted 1600 tokens and the target accepted 1195. Result notes. Median decode of 5 samples between 47.585 and 48.288 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-confirm-1gpu-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T07:34:04Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 20.2 tok/s Decode 47.9 tok/s Time to first token 543.8 ms Baseline q38n0 (+42.6%) Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the prose prompt; samples spanned 1.5% of the minimum. The head drafted 1600 tokens and the target accepted 1195. Result notes. Median decode of 5 samples between 47.496 and 48.203 tok/s. Every sample matched the matched non-speculative arm. `qwen38-confirm-1gpu-mtp2-strict-json-20260817` Task text generation Status VALID Measured 2026-08-17T07:34:04Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 24 Prompt tokens 24 Generated tokens 22 Prefill 49.1 tok/s Decode 51.8 tok/s Time to first token 488.4 ms Baseline q38n0 (+53.3%) Output matches baseline yes Evidence qwen38/results/20260817T073052Z-qwen38-27b-mtp-confirm-single Benchmark notes. Retained confirmation of the selected profile. Median of 5 samples on the strict JSON task; samples spanned 6.3% of the minimum. The head drafted 80 tokens and the target accepted 65. Result notes. Median decode of 5 samples between 48.928 and 52.022 tok/s. Every sample matched the matched non-speculative arm. `qwen38-len1024-1gpu-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T07:03:40Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 1024 Prefill 73.5 tok/s Decode 50.7 tok/s Time to first token 462.8 ms Baseline q38n0 (+51.9%) Output matches baseline no Evidence qwen38/results/20260817T065907Z-qwen38-27b-mtp-length-1024 Benchmark notes. Generated-length series at 1,024 tokens on the b10358 build. Median of 3 samples on the code prompt; samples spanned 0.7% of the minimum. The head drafted 2310 tokens and the target accepted 1911. Result notes. Median decode of 3 samples between 50.601 and 50.976 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-len1024-1gpu-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T07:03:40Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 1024 Prefill 24.7 tok/s Decode 49.7 tok/s Time to first token 445.0 ms Baseline q38n0 (+49.9%) Output matches baseline yes Evidence qwen38/results/20260817T065907Z-qwen38-27b-mtp-length-1024 Benchmark notes. Generated-length series at 1,024 tokens on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 0.7% of the minimum. The head drafted 2337 tokens and the target accepted 1899. Result notes. Median decode of 3 samples between 49.722 and 50.072 tok/s. Every sample matched the matched non-speculative arm. `qwen38-len128-1gpu-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:57:26Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 128 Prefill 69.2 tok/s Decode 48.7 tok/s Time to first token 491.0 ms Baseline q38n0 (+42.6%) Output matches baseline no Evidence qwen38/results/20260817T065631Z-qwen38-27b-mtp-length-128 Benchmark notes. Generated-length series at 128 tokens on the b10358 build. Median of 3 samples on the code prompt; samples spanned 3.1% of the minimum. The head drafted 294 tokens and the target accepted 231. Result notes. Median decode of 3 samples between 48.412 and 49.919 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-len128-1gpu-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:57:26Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 128 Prefill 21.1 tok/s Decode 46.6 tok/s Time to first token 521.2 ms Baseline q38n0 (+36.4%) Output matches baseline yes Evidence qwen38/results/20260817T065631Z-qwen38-27b-mtp-length-128 Benchmark notes. Generated-length series at 128 tokens on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 0.6% of the minimum. The head drafted 318 tokens and the target accepted 219. Result notes. Median decode of 3 samples between 46.525 and 46.813 tok/s. Every sample matched the matched non-speculative arm. `qwen38-width-1gpu-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:43:34Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 59.2 tok/s Decode 47.7 tok/s Time to first token 574.6 ms Baseline q38n0 (+42.2%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 1.9% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 47.383 and 48.277 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-1gpu-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:43:34Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 23.3 tok/s Decode 48.1 tok/s Time to first token 471.6 ms Baseline q38n0 (+43.0%) Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 1.8% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 47.608 and 48.473 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	4096	11	11	Local GPU DFlash	+30.7%	69.7%	—	43.4	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU DFlash	+39.8%	78.7%	—	46.4	NA	NA
`q38d22` Qwen3.8 27B Q4_K_M on one V100 at a 4,096-token context with batch and microbatch 512, Flash Attention, F16 KV, and DFlash2 at width two.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama5ecbe1ac1-1v100-b512-c4k-f16-fa-d2n2 Created 2026-08-20T06:39:50Z Notes Served by the DFlash2 llama.cpp build so that every arm of this experiment shares one runtime. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.spec_draft_p_min 0.0 — Draft probability floor passed to llama-server. system.device_memory_used 19054 MiB MiB — Reported by nvidia-smi after the model loaded. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; One V100 with --split-mode none. Results and evidence `qwen38-dflash2-d2n2-code-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Decode 46.4 tok/s Baseline q38d2o (+39.5%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The code prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.8% of the minimum. Result notes. Decode median of five samples on the code prompt. `qwen38-dflash2-d2n2-prose-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Decode 43.4 tok/s Baseline q38d2o (+30.7%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The prose prompt at 400 generated tokens. Median of 5 samples; the samples spanned 1.1% of the minimum. Result notes. Decode median of five samples on the prose prompt.
	llama.cpp	V100	4096	11	11	Local GPU DFlash	+16.1%	47.4%	—	38.5	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU DFlash	+38.4%	62.0%	—	45.9	NA	NA
`q38d24` Qwen3.8 27B Q4_K_M on one V100 at a 4,096-token context with batch and microbatch 512, Flash Attention, F16 KV, and DFlash2 at width four.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama5ecbe1ac1-1v100-b512-c4k-f16-fa-d2n4 Created 2026-08-20T06:39:50Z Notes Served by the DFlash2 llama.cpp build so that every arm of this experiment shares one runtime. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.spec_draft_p_min 0.0 — Draft probability floor passed to llama-server. system.device_memory_used 19354 MiB MiB — Reported by nvidia-smi after the model loaded. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; One V100 with --split-mode none. Results and evidence `qwen38-dflash2-d2n4-code-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Decode 45.9 tok/s Baseline q38d2o (+38.1%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The code prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.9% of the minimum. Result notes. Decode median of five samples on the code prompt. `qwen38-dflash2-d2n4-prose-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Decode 38.5 tok/s Baseline q38d2o (+16.1%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The prose prompt at 400 generated tokens. Median of 5 samples; the samples spanned 1.1% of the minimum. Result notes. Decode median of five samples on the prose prompt.
	llama.cpp	V100	4096	11	11	Local GPU DFlash	-13.3%	33.1%	—	28.8	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU DFlash	+8.2%	44.9%	—	35.9	NA	NA
`q38d27` Qwen3.8 27B Q4_K_M on one V100 at a 4,096-token context with batch and microbatch 512, Flash Attention, F16 KV, and DFlash2 at width seven.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama5ecbe1ac1-1v100-b512-c4k-f16-fa-d2n7 Created 2026-08-20T06:39:50Z Notes Served by the DFlash2 llama.cpp build so that every arm of this experiment shares one runtime. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.spec_draft_p_min 0.0 — Draft probability floor passed to llama-server. system.device_memory_used 19802 MiB MiB — Reported by nvidia-smi after the model loaded. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; One V100 with --split-mode none. Results and evidence `qwen38-dflash2-d2n7-code-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Decode 35.9 tok/s Baseline q38d2o (+8.0%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The code prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.3% of the minimum. Result notes. Decode median of five samples on the code prompt. `qwen38-dflash2-d2n7-prose-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Decode 28.8 tok/s Baseline q38d2o (-13.3%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The prose prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.7% of the minimum. Result notes. Decode median of five samples on the prose prompt.
	llama.cpp	V100	4096	11	11	Local GPU DFlash	-13.1%	33.1%	—	28.8	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU DFlash	+7.7%	44.9%	—	35.7	NA	NA
`q38d2p` Qwen3.8 27B Q4_K_M on one V100 at a 4,096-token context with batch and microbatch 512, Flash Attention, F16 KV, and DFlash2 at width seven with a 0.75 probability floor.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama5ecbe1ac1-1v100-b512-c4k-f16-fa-d2n7p75 Created 2026-08-20T06:39:50Z Notes Served by the DFlash2 llama.cpp build so that every arm of this experiment shares one runtime. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.spec_draft_p_min 0.75 — Draft probability floor passed to llama-server. system.device_memory_used 19802 MiB MiB — Reported by nvidia-smi after the model loaded. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; One V100 with --split-mode none. Results and evidence `qwen38-dflash2-d2n7p75-code-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Decode 35.7 tok/s Baseline q38d2o (+7.4%) Evidence qwen38/results/20260820T065543Z-qwen38-27b-dflash2-pmin Benchmark notes. The code prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.7% of the minimum. Result notes. Decode median of five samples on the code prompt. `qwen38-dflash2-d2n7p75-prose-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Decode 28.8 tok/s Baseline q38d2o (-13.1%) Evidence qwen38/results/20260820T065543Z-qwen38-27b-dflash2-pmin Benchmark notes. The prose prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.6% of the minimum. Result notes. Decode median of five samples on the prose prompt.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+47.0%	74.7%	—	48.8	NA	NA
	llama.cpp	V100	4096	34	34	Local GPU MTP	+46.2%	74.7%	—	48.5	NA	NA
`q38d2m` Qwen3.8 27B Q4_K_M on one V100 at a 4,096-token context with batch and microbatch 512, Flash Attention, F16 KV, and the model native MTP head at width two.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama5ecbe1ac1-1v100-b512-c4k-f16-fa-mtp2 Created 2026-08-20T06:39:50Z Notes Served by the DFlash2 llama.cpp build so that every arm of this experiment shares one runtime. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.spec_draft_p_min 0.0 — Draft probability floor passed to llama-server. system.device_memory_used 16928 MiB MiB — Reported by nvidia-smi after the model loaded. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; One V100 with --split-mode none. Results and evidence `qwen38-dflash2-mtp2-code-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Decode 48.5 tok/s Baseline q38d2o (+45.9%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The code prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.6% of the minimum. Result notes. Decode median of five samples on the code prompt. `qwen38-dflash2-mtp2-prose-20260820` Task text generation Status VALID Measured 2026-08-20T06:39:50Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Decode 48.8 tok/s Baseline q38d2o (+47.0%) Evidence qwen38/results/20260820T063950Z-qwen38-27b-dflash2-single Benchmark notes. The prose prompt at 400 generated tokens. Median of 5 samples; the samples spanned 0.7% of the minimum. Result notes. Decode median of five samples on the prose prompt.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+27.2%	68.9%	70.1	41.6	NA	NA
	llama.cpp	V100	4096	11	11	Local GPU MTP	+32.4%	74.7%	69.6	43.3	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+37.2%	78.6%	184.2	44.8	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+23.8%	74.7%	181.9	40.4	NA	NA
`q38l2` Qwen3.8 27B Q4_K_M split 1:1 with native MTP width two, b9190.4 results · 4 techniques · 2 parameters Configuration ID qwen38-llamab9190-2xv100-layer-mtp2-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Same-host layer split One process with a 1:1 layer split across both V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-repro-2gpu-layer-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T07:38:52Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 128 Prefill 184.2 tok/s Decode 44.8 tok/s Time to first token 184.6 ms Baseline q38l0 (+36.2%) Output matches baseline no Evidence qwen38/results/20260817T073738Z-qwen38-27b-mtp-repro-b9190-128 Benchmark notes. Reproduction of the invalid 2026-08-14 arm under its own build, placement, and generated length. Median of 5 samples on the code prompt; samples spanned 0.7% of the minimum. The head drafted 490 tokens and the target accepted 385. Result notes. Median decode of 5 samples between 44.612 and 44.918 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-repro-2gpu-layer-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T07:38:52Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 128 Prefill 70.1 tok/s Decode 41.6 tok/s Time to first token 156.9 ms Baseline q38l0 (+27.2%) Output matches baseline yes Evidence qwen38/results/20260817T073738Z-qwen38-27b-mtp-repro-b9190-128 Benchmark notes. Reproduction of the invalid 2026-08-14 arm under its own build, placement, and generated length. Median of 5 samples on the prose prompt; samples spanned 0.7% of the minimum. The head drafted 530 tokens and the target accepted 365. Result notes. Median decode of 5 samples between 41.285 and 41.582 tok/s. Every sample matched the matched non-speculative arm. `qwen38-screen-2gpu-layer-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:31:54Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 181.9 tok/s Decode 40.4 tok/s Time to first token 186.9 ms Baseline q38l0 (+22.4%) Output matches baseline no Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the code prompt; samples spanned 14.7% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 37.672 and 43.200 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-screen-2gpu-layer-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:31:54Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 69.6 tok/s Decode 43.3 tok/s Time to first token 158.0 ms Baseline q38l0 (+31.2%) Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the prose prompt; samples spanned 0.4% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 43.227 and 43.400 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+28.1%	74.7%	65.6	42.4	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+27.1%	74.7%	178.3	42.0	NA	NA
`q38s2` Qwen3.8 27B Q4_K_M on one V100 with native MTP width two, b9190.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab9190-v100-single-mtp2-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-screen-1gpu-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:28:27Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 178.3 tok/s Decode 42.0 tok/s Time to first token 190.7 ms Baseline q38s0 (+28.0%) Output matches baseline no Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the code prompt; samples spanned 0.1% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 42.003 and 42.063 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-screen-1gpu-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:28:27Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 65.6 tok/s Decode 42.4 tok/s Time to first token 167.6 ms Baseline q38s0 (+28.1%) Output matches baseline yes Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the prose prompt; samples spanned 1.0% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 42.140 and 42.551 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	4096	11	11	Local GPU MTP	-18.2%	73.1%	25.6	26.1	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	-18.8%	73.4%	93.3	25.9	NA	NA
`q38t2` Qwen3.8 27B Q4_K_M tensor split with native MTP width two, b9190.2 results · 5 techniques · 2 parameters Configuration ID qwen38-llamab9190-2xv100-tensor-mtp2-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. GPU link topology Every all-reduce crossed the SYS path between the cards. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Tensor parallelism One process with a tensor split across both V100s. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-screen-2gpu-tensor-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:35:47Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 93.3 tok/s Decode 25.9 tok/s Time to first token 364.3 ms Baseline q38t0 (-14.9%) Output matches baseline no Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the code prompt; samples spanned 18.5% of the minimum. The head drafted 969 tokens and the target accepted 711. Result notes. Median decode of 3 samples between 24.800 and 29.387 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-screen-2gpu-tensor-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:35:47Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 25.6 tok/s Decode 26.1 tok/s Time to first token 429.4 ms Baseline q38t0 (-18.2%) Output matches baseline no Evidence qwen38/results/20260817T062621Z-qwen38-27b-mtp-placement-screen Benchmark notes. Placement screen on the pinned b9190 build. Median of 3 samples on the prose prompt; samples spanned 2.4% of the minimum. The head drafted 972 tokens and the target accepted 711. Result notes. Median decode of 3 samples between 25.883 and 26.494 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+38.4%	87.3%	19.6	46.5	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+38.1%	89.1%	64.4	46.4	NA	NA
`q38n1` Qwen3.8 27B Q4_K_M on one V100 with native MTP width one, b10358.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp1-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 1 tokens per draft call. Speculative policy tuning Draft width 1 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-width-1gpu-mtp1-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:42:05Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 64.4 tok/s Decode 46.4 tok/s Time to first token 528.2 ms Baseline q38n0 (+38.3%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 0.3% of the minimum. The head drafted 633 tokens and the target accepted 564. Result notes. Median decode of 3 samples between 46.278 and 46.414 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-1gpu-mtp1-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:42:05Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 19.6 tok/s Decode 46.5 tok/s Time to first token 561.0 ms Baseline q38n0 (+38.4%) Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 3.3% of the minimum. The head drafted 639 tokens and the target accepted 558. Result notes. Median decode of 3 samples between 45.241 and 46.711 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+30.6%	62.0%	30.5	43.9	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+35.4%	64.9%	72.0	45.5	NA	NA
`q38n3` Qwen3.8 27B Q4_K_M on one V100 with native MTP width three, b10358.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp3-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 3 tokens per draft call. Speculative policy tuning Draft width 3 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-width-1gpu-mtp3-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:45:00Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 72.0 tok/s Decode 45.5 tok/s Time to first token 472.3 ms Baseline q38n0 (+35.6%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 1.3% of the minimum. The head drafted 1215 tokens and the target accepted 789. Result notes. Median decode of 3 samples between 45.281 and 45.856 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-1gpu-mtp3-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:45:00Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 30.5 tok/s Decode 43.9 tok/s Time to first token 360.7 ms Baseline q38n0 (+30.6%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 2.3% of the minimum. The head drafted 1254 tokens and the target accepted 777. Result notes. Median decode of 3 samples between 42.924 and 43.904 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+24.3%	69.3%	25.5	41.8	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+32.9%	73.7%	63.5	44.7	NA	NA
`q38ng` Qwen3.8 27B Q4_K_M on one V100 with native MTP width four gated at a 0.60 minimum draft probability, b10358.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp4-p60-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 4 tokens per draft call. Speculative policy tuning Draft width 4 with a 0.60 minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-width-1gpu-mtp4-p60-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:49:53Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 63.5 tok/s Decode 44.7 tok/s Time to first token 535.8 ms Baseline q38n0 (+33.2%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 4.5% of the minimum. The head drafted 1083 tokens and the target accepted 798. Result notes. Median decode of 3 samples between 43.383 and 45.347 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-1gpu-mtp4-p60-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:49:53Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 25.5 tok/s Decode 41.8 tok/s Time to first token 430.7 ms Baseline q38n0 (+24.2%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 1.1% of the minimum. The head drafted 1085 tokens and the target accepted 752. Result notes. Median decode of 3 samples between 41.373 and 41.837 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+31.3%	57.6%	28.6	44.1	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+28.8%	57.0%	76.5	43.3	NA	NA
`q38n4` Qwen3.8 27B Q4_K_M on one V100 with native MTP width four, b10358.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp4-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 4 tokens per draft call. Speculative policy tuning Draft width 4 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-width-1gpu-mtp4-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:46:31Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 76.5 tok/s Decode 43.3 tok/s Time to first token 444.5 ms Baseline q38n0 (+29.1%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 1.3% of the minimum. The head drafted 1458 tokens and the target accepted 831. Result notes. Median decode of 3 samples between 43.106 and 43.667 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-1gpu-mtp4-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:46:31Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 28.6 tok/s Decode 44.1 tok/s Time to first token 384.5 ms Baseline q38n0 (+31.2%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 1.4% of the minimum. The head drafted 1449 tokens and the target accepted 834. Result notes. Median decode of 3 samples between 43.558 and 44.183 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+1.1%	43.0%	20.0	34.0	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+7.2%	46.5%	56.9	36.0	NA	NA
`q38n6` Qwen3.8 27B Q4_K_M on one V100 with native MTP width six, b10358.2 results · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp6-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 6 tokens per draft call. Speculative policy tuning Draft width 6 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-width-1gpu-mtp6-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:48:02Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 56.9 tok/s Decode 36.0 tok/s Time to first token 597.1 ms Baseline q38n0 (+7.4%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 0.4% of the minimum. The head drafted 1890 tokens and the target accepted 879. Result notes. Median decode of 3 samples between 35.934 and 36.081 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-1gpu-mtp6-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:48:02Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 20.0 tok/s Decode 34.0 tok/s Time to first token 550.3 ms Baseline q38n0 (+1.0%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 0.6% of the minimum. The head drafted 2001 tokens and the target accepted 861. Result notes. Median decode of 3 samples between 33.868 and 34.056 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run.
	llama.cpp	V100	4096	11	11	Local GPU MTP	+43.3%	74.7%	19.7	49.1	NA	NA
	llama.cpp	V100(2)	4096	34	34	Local GPU MTP	+29.1%	74.7%	73.3	44.2	NA	NA
`q38d2` Qwen3.8 27B Q4_K_M split 1:1 with native MTP width two, b10358.2 results · 4 techniques · 2 parameters Configuration ID qwen38-llamab10358-2xv100-layer-mtp2-p0-b512-c4k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Same-host layer split One process with a 1:1 layer split across both V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-width-2gpu-layer-mtp2-code-20260817` Task text generation Status VALID Measured 2026-08-17T06:53:17Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 34 Prompt tokens 34 Generated tokens 400 Prefill 73.3 tok/s Decode 44.2 tok/s Time to first token 463.8 ms Baseline q38d0 (+29.4%) Output matches baseline no Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the code prompt; samples spanned 26.8% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 38.473 and 48.782 tok/s. The greedy sequence diverged from the matched non-speculative arm of the same run. `qwen38-width-2gpu-layer-mtp2-prose-20260817` Task text generation Status VALID Measured 2026-08-17T06:53:17Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 11 Prompt tokens 11 Generated tokens 400 Prefill 19.7 tok/s Decode 49.1 tok/s Time to first token 557.0 ms Baseline q38d0 (+43.3%) Output matches baseline yes Evidence qwen38/results/20260817T064004Z-qwen38-27b-mtp-width-b10358 Benchmark notes. Draft-width sweep on the b10358 build. Median of 3 samples on the prose prompt; samples spanned 0.2% of the minimum. The head drafted 960 tokens and the target accepted 717. Result notes. Median decode of 3 samples between 49.014 and 49.131 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	40960	30906	30906	Local GPU MTP	+63.6%	88.1%	552.5	45.6	24%	NA
`q38p2` Qwen3.8 27B Q4_K_M on one V100 at a 40,960-token context with native MTP width two, b10358.1 result · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp2-p0-b512-c40k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-depth-1gpu-mtp2-depth-20260817` Task text generation Status VALID Measured 2026-08-17T07:17:09Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 552.5 tok/s Decode 45.6 tok/s Time to first token 55937.3 ms Baseline q38p0 (+63.6%) Output matches baseline yes Evidence qwen38/results/20260817T071250Z-qwen38-27b-mtp-depth-32k Benchmark notes. Deep-context pair. The prompt is 100,000 bytes of this repository's tracked Markdown in a fixed order. Median of 3 samples on the deep-context prompt; samples spanned 1.5% of the minimum. The head drafted 429 tokens and the target accepted 378. Result notes. Median decode of 3 samples between 45.066 and 45.750 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100(2)	40960	30906	30906	Local GPU MTP	+65.8%	88.1%	543.2	46.4	24%	NA
`q38q2` Qwen3.8 27B Q4_K_M split 1:1 at a 40,960-token context with native MTP width two, b10358.1 result · 4 techniques · 2 parameters Configuration ID qwen38-llamab10358-2xv100-layer-mtp2-p0-b512-c40k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Same-host layer split One process with a 1:1 layer split across both V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; NUMA interleave=all; Physical GPU 1. Results and evidence `qwen38-depth-2gpu-layer-mtp2-depth-20260817` Task text generation Status VALID Measured 2026-08-17T07:25:31Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 40960 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 543.2 tok/s Decode 46.4 tok/s Time to first token 56897.4 ms Baseline q38q0 (+65.8%) Output matches baseline yes Evidence qwen38/results/20260817T071250Z-qwen38-27b-mtp-depth-32k Benchmark notes. Deep-context pair. The prompt is 100,000 bytes of this repository's tracked Markdown in a fixed order. Median of 3 samples on the deep-context prompt; samples spanned 11.7% of the minimum. The head drafted 429 tokens and the target accepted 378. Result notes. Median decode of 3 samples between 41.598 and 46.445 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	32768	30906	30906	Local GPU MTP	+62.7%	88.1%	552.9	45.2	24%	NA
`q38v2` Qwen3.8 27B Q4_K_M on one V100 at the 32,768-token router context with native MTP width two, b10358.1 result · 3 techniques · 2 parameters Configuration ID qwen38-llamab10358-v100-single-mtp2-p0-b512-c32k-f16-fa Created 2026-08-17T06:26:21Z Notes One llama.cpp server process. Automatic fit is off, and every placement and speculative setting is explicit. Applied optimizations GPU layer offload The complete target ran on the V100s. Speculative decoding The built-in MTP head drafted 2 tokens per draft call. Speculative policy tuning Draft width 2 with a zero minimum draft probability. Uncommon parameters target.artifact_revision 4604b899a826000505a834e623272db5b7fd62f6 — Pinned unsloth/Qwen3.8-27B-GGUF revision. target.prefix_cache disabled — Every timed request set cache_prompt false, so each one processed its complete prompt. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0. Results and evidence `qwen38-serving-1gpu-mtp2-depth-20260817` Task text generation Status VALID Measured 2026-08-17T07:46:06Z Statistic median; 3 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 30906 Prompt tokens 30906 Generated tokens 200 Prefill 552.9 tok/s Decode 45.2 tok/s Time to first token 55898.8 ms Baseline q38v0 (+62.7%) Output matches baseline yes Evidence qwen38/results/20260817T074146Z-qwen38-27b-mtp-serving-32k Benchmark notes. Serving-profile check at the router context and depth. Median of 3 samples on the deep-context prompt; samples spanned 1.0% of the minimum. The head drafted 429 tokens and the target accepted 378. Result notes. Median decode of 3 samples between 44.987 and 45.424 tok/s. Every sample matched the matched non-speculative arm.
	llama.cpp	V100	131072	111781	111781	Local GPU MTP	-11.3%	—	429.4	24.8	19%	NA
	llama.cpp	V100	131072	112941	112941	Local GPU MTP	+17.6%	—	427.9	32.9	19%	NA
	llama.cpp	V100	131072	116943	116943	Local GPU MTP	-0.3%	—	420.0	27.9	18%	NA
	llama.cpp	V100	131072	118103	118103	Local GPU MTP	-13.8%	—	418.4	24.1	18%	NA
`q38cr0` Qwen3.8 27B Q4_K_M on one V100 at a 131,072-token context with microbatch 4096, the MTP head at width two, and --cache-ram 0 MiB.4 results · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b4096-c128k-f16-fa-mtp2-cram0 Created 2026-08-19T02:31:51Z Notes The host prompt cache is disabled. The server reported a prompt state of 8,739 MiB for one session of about 112,000 tokens. Applied optimizations No applied technique is cataloged. Uncommon parameters system.host_prompt_cache disabled MiB — --cache-ram value for the arm. system.model_cache_state warm — The artifact was resident in the host page cache. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. system.retained_prefixes 0 prefixes — Whole 112,000-token prefixes the budget can hold at the measured 8,739 MiB each. system.server_slots 1 slots — --parallel 1; the router serves one slot. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-cache-cram0-a1-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 429.4 tok/s Decode 24.8 tok/s Time to first token 260314.4 ms Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session A first request; a cold prefill in every arm. --cache-ram 0 MiB. The timed request was a prefix-cache miss: 0 of 111781 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.3% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 259698 ms, maximum 260587 ms. 0 of 111781 prompt tokens came from cache. `qwen38-cache-cram0-a2-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 112941 Prompt tokens 112941 Generated tokens 32 Prefill 427.9 tok/s Decode 32.9 tok/s Time to first token 263915.5 ms Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session A returns with one appended turn. --cache-ram 0 MiB. The timed request was a prefix-cache miss: 0 of 112941 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.2% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 263684 ms, maximum 264296 ms. 0 of 112941 prompt tokens came from cache. `qwen38-cache-cram0-b1-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 116943 Prompt tokens 116943 Generated tokens 32 Prefill 420.0 tok/s Decode 27.9 tok/s Time to first token 278434.2 ms Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session B first request; takes the slot from session A. --cache-ram 0 MiB. The timed request was a prefix-cache miss: 0 of 116943 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.2% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 278063 ms, maximum 278636 ms. 0 of 116943 prompt tokens came from cache. `qwen38-cache-cram0-b2-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 118103 Prompt tokens 118103 Generated tokens 32 Prefill 418.4 tok/s Decode 24.1 tok/s Time to first token 282258.2 ms Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session B returns with one appended turn. --cache-ram 0 MiB. The timed request was a prefix-cache miss: 0 of 118103 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.2% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 282085 ms, maximum 282615 ms. 0 of 118103 prompt tokens came from cache.
	llama.cpp	V100	131072	111781	111781	Local GPU MTP	+4.2%	—	430.0	25.9	19%	NA
	llama.cpp	V100	131072	112941	112941	Local GPU MTP	+33.0%	—	428.5	33.0	19%	NA
	llama.cpp	V100	131072	116943	116943	Local GPU MTP	+12.6%	—	419.9	27.9	18%	NA
	llama.cpp	V100	131072	118103	118103	Local GPU MTP	-2.5%	—	418.3	24.2	18%	NA
`q38cr8` Qwen3.8 27B Q4_K_M on one V100 at a 131,072-token context with microbatch 4096, the MTP head at width two, and --cache-ram 8192 MiB.4 results · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b4096-c128k-f16-fa-mtp2-cram8192 Created 2026-08-19T02:31:51Z Notes The llama-server default, and what the router ran until 2026-08-20. The server reported a prompt state of 8,739 MiB for one session of about 112,000 tokens. Applied optimizations No applied technique is cataloged. Uncommon parameters system.host_prompt_cache enabled MiB — --cache-ram value for the arm. system.model_cache_state warm — The artifact was resident in the host page cache. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. system.retained_prefixes 0 prefixes — Whole 112,000-token prefixes the budget can hold at the measured 8,739 MiB each. system.server_slots 1 slots — --parallel 1; the router serves one slot. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-cache-cram8192-a1-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 430.0 tok/s Decode 25.9 tok/s Time to first token 259946.6 ms Baseline q38cr0 (+0.1%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session A first request; a cold prefill in every arm. --cache-ram 8192 MiB. The timed request was a prefix-cache miss: 0 of 111781 prompt tokens were already processed. Median of 5 samples; time to first token spanned 11.5% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 259752 ms, maximum 289745 ms. 0 of 111781 prompt tokens came from cache. `qwen38-cache-cram8192-a2-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 112941 Prompt tokens 112941 Generated tokens 32 Prefill 428.5 tok/s Decode 33.0 tok/s Time to first token 263565.9 ms Baseline q38cr0 (+0.1%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session A returns with one appended turn. --cache-ram 8192 MiB. The timed request was a prefix-cache miss: 0 of 112941 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.2% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 263421 ms, maximum 263833 ms. 0 of 112941 prompt tokens came from cache. `qwen38-cache-cram8192-b1-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 116943 Prompt tokens 116943 Generated tokens 32 Prefill 419.9 tok/s Decode 27.9 tok/s Time to first token 278494.2 ms Baseline q38cr0 (-0.0%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session B first request; takes the slot from session A. --cache-ram 8192 MiB. The timed request was a prefix-cache miss: 0 of 116943 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.4% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 277794 ms, maximum 278890 ms. 0 of 116943 prompt tokens came from cache. `qwen38-cache-cram8192-b2-20260819` Task text generation Status VALID Measured 2026-08-19T02:31:51Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 118103 Prompt tokens 118103 Generated tokens 32 Prefill 418.3 tok/s Decode 24.2 tok/s Time to first token 282316.8 ms Baseline q38cr0 (-0.0%) Evidence qwen38/results/20260819T023151Z-qwen38-27b-prompt-cache Benchmark notes. Session B returns with one appended turn. --cache-ram 8192 MiB. The timed request was a prefix-cache miss: 0 of 118103 prompt tokens were already processed. Median of 5 samples; time to first token spanned 0.1% of the minimum. Result notes. Time to first token is the server prompt time. Minimum 282284 ms, maximum 282439 ms. 0 of 118103 prompt tokens came from cache.
	llama.cpp	V100	131072	111781	111781	Local CPU MTP	-2.0%	—	423.6	29.0	19%	NA
`q38dp2` Qwen3.8 27B Q4_K_M on one V100 with microbatch 2048 and logical batch 2048.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b2048-c128k-f16-fa-mtp2-depth112k Created 2026-08-19T09:41:59Z Notes Cold prefill of a 111,781-token prompt. The server reported 26894 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 26894 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth112k-ub2048-20260819` Task text generation Status VALID Measured 2026-08-19T09:41:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 423.6 tok/s Decode 29.0 tok/s Baseline q38dp5 (+15.8%) Output matches baseline no Evidence qwen38/results/20260819T094159Z-qwen38-27b-depth112k-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 2048 and the production context. Samples spanned 0.1% of the minimum, from 423.5 to 424.0 tokens per second. That is 1.158 times the 512 arm. Result notes. Cold prefill at a 111,781-token prompt with the host prompt cache disabled. Minimum 423.5, maximum 424.0. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	131072	111781	111781	Local CPU MTP	-3.8%	—	430.1	28.5	19%	NA
`q38dp4` Qwen3.8 27B Q4_K_M on one V100 with microbatch 4096 and logical batch 4096.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b4096-c128k-f16-fa-mtp2-depth112k Created 2026-08-19T09:41:59Z Notes Cold prefill of a 111,781-token prompt. The server reported 28680 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 28680 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth112k-ub4096-20260819` Task text generation Status VALID Measured 2026-08-19T09:41:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 430.1 tok/s Decode 28.5 tok/s Baseline q38dp5 (+17.6%) Output matches baseline yes Evidence qwen38/results/20260819T094159Z-qwen38-27b-depth112k-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 4096 and the production context. Samples spanned 0.1% of the minimum, from 429.9 to 430.5 tokens per second. That is 1.176 times the 512 arm. Result notes. Cold prefill at a 111,781-token prompt with the host prompt cache disabled. Minimum 429.9, maximum 430.5. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	131072	111781	111781	Local CPU MTP	+5.7%	—	365.9	29.6	16%	NA
`q38dp5` Qwen3.8 27B Q4_K_M on one V100 with microbatch 512 and logical batch 2048.1 result · 0 techniques · 5 parameters Configuration ID qwen38-llama9d77fa172-1v100-b512-c128k-f16-fa-mtp2-depth112k Created 2026-08-19T09:41:59Z Notes Cold prefill of a 111,781-token prompt. The server reported 25554 MiB of device memory in use. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.cuda_graphs enabled — b10488 captures CUDA graphs on Volta. system.device_memory_used 25554 MiB MiB — Reported by nvidia-smi after the model loaded. system.host_prompt_cache disabled MiB — --cache-ram 0; every sample paid the whole prefill. system.prefix_cache_state miss — cache_prompt false; a cold prefill in every sample. system.resident_gpu_services unloaded — lab-gpu-vram-eaters.target was stopped for the run. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Physical GPU 0 on the x8 root port. Results and evidence `qwen38-depth112k-ub512-20260819` Task text generation Status VALID Measured 2026-08-19T09:41:59Z Statistic median; 5 samples Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 365.9 tok/s Decode 29.6 tok/s Output matches baseline yes Evidence qwen38/results/20260819T094159Z-qwen38-27b-depth112k-ubatch Benchmark notes. Median of five cold-prefill samples at microbatch 512 and the production context. Samples spanned 0.1% of the minimum, from 365.7 to 366.0 tokens per second. That is 1.000 times the 512 arm. Result notes. Cold prefill at a 111,781-token prompt with the host prompt cache disabled. Minimum 365.7, maximum 366.0. Decode is reported for completeness; this benchmark measures prefill.
	llama.cpp	V100	131072	111781	111781	Local GPU MTP	—	69.2%	417.8	—	18%	NA
	llama.cpp	V100	131072	116943	116943	Local GPU MTP	—	72.0%	412.9	—	18%	NA
`q38rtm` Qwen3.8 27B Q4_K_M served by the deployed r720 router on one V100 at a 131,072-token context with microbatch 2048, the MTP head at width two, and --cache-ram 65536 MiB. Cold prefix.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama030ebb558-1v100-b2048-c128k-f16-fa-mtp2-cram65536-miss Created 2026-08-20T05:41:00Z Notes The deployed serving preset. The timed request found nothing in the host prompt cache and processed its whole prompt. Applied optimizations No applied technique is cataloged. Uncommon parameters system.host_prompt_cache 65536 MiB MiB — --cache-ram 65536; large enough to store both session prefixes. system.prefix_cache_state miss — cache_prompt true, but the server had never seen this prefix. system.resident_gpu_services resident — The normal service set stayed up; this is the production path. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Served by the deployed local-ai-router preset. Results and evidence `qwen38-router-cache-a1-20260820` Task text generation Status PROVISIONAL Measured 2026-08-20T05:41:00Z Statistic single; 1 sample Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 111781 Prompt tokens 111781 Generated tokens 32 Prefill 417.8 tok/s Time to first token 267576.6 ms Evidence qwen38/results/20260820T054100Z-qwen38-27b-router-cache-verify Benchmark notes. Session A first request through the deployed router; a cold prefill. The timed request was a prefix-cache miss: 0 of 111781 prompt tokens were already processed. One retained sample; an earlier identical replay agreed within 0.5%. Result notes. Time to first token is the server prompt time. 0 of 111781 prompt tokens came from cache. An earlier identical replay measured 266274.4 ms, so the two agree within 0.5%. Decode is not recorded: each request generated 32 tokens, too few to measure it. `qwen38-router-cache-b1-20260820` Task text generation Status PROVISIONAL Measured 2026-08-20T05:41:00Z Statistic single; 1 sample Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 116943 Prompt tokens 116943 Generated tokens 32 Prefill 412.9 tok/s Time to first token 283232.3 ms Evidence qwen38/results/20260820T054100Z-qwen38-27b-router-cache-verify Benchmark notes. Session B first request; a cold prefill that took the slot from session A. The timed request was a prefix-cache miss: 0 of 116943 prompt tokens were already processed. One retained sample; an earlier identical replay agreed within 0.3%. Result notes. Time to first token is the server prompt time. 0 of 116943 prompt tokens came from cache. An earlier identical replay measured 282591.6 ms, so the two agree within 0.3%. Decode is not recorded: each request generated 32 tokens, too few to measure it.
	llama.cpp	V100	131072	112941	112941	Local GPU MTP	—	90.9%	226.3	—	10%	NA
	llama.cpp	V100	131072	118103	118103	Local GPU MTP	—	55.2%	223.2	—	10%	NA
`q38rth` Qwen3.8 27B Q4_K_M served by the deployed r720 router on one V100 at a 131,072-token context with microbatch 2048, the MTP head at width two, and --cache-ram 65536 MiB. Restored prefix.2 results · 0 techniques · 3 parameters Configuration ID qwen38-llama030ebb558-1v100-b2048-c128k-f16-fa-mtp2-cram65536-hit Created 2026-08-20T05:41:00Z Notes The deployed serving preset. The timed request restored its prefix from the host prompt cache and processed only the remainder. Applied optimizations No applied technique is cataloged. Uncommon parameters system.host_prompt_cache 65536 MiB MiB — --cache-ram 65536; large enough to store both session prefixes. system.prefix_cache_state hit — The session returned after another session took the slot. The reusable prefix is truncated to a whole microbatch, so 3,208 tokens remained. system.resident_gpu_services resident — The normal service set stayed up; this is the production path. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 65 target layers; 20 threads; 20 batch threads; affinity all 40 logical CPUs; NUMA interleave=all; Served by the deployed local-ai-router preset. Results and evidence `qwen38-router-cache-a2-20260820` Task text generation Status PROVISIONAL Measured 2026-08-20T05:41:00Z Statistic single; 1 sample Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 112941 Prompt tokens 112941 Generated tokens 32 Prefill 226.3 tok/s Time to first token 14177.5 ms Baseline q38rtm (+1787.3%) Evidence qwen38/results/20260820T054100Z-qwen38-27b-router-cache-verify Benchmark notes. Session A returns with one appended turn. The timed request was a prefix-cache hit: 109733 of 112941 prompt tokens were restored from the host cache and 3208 were processed. One retained sample; an earlier identical replay agreed within 1.8%. Result notes. Time to first token is the server prompt time, 18.87 times lower than the 267576.6 ms cold first visit of the same session. 109733 of 112941 prompt tokens were restored and 3208 processed. Prefill throughput is over the 3208 processed tokens only, so it is far below the cold figure because most of the 14.2 s restores cache state. An earlier identical replay measured 13929.0 ms, so the two agree within 1.8%. Decode is not recorded: each request generated 32 tokens, too few to measure it. `qwen38-router-cache-b2-20260820` Task text generation Status PROVISIONAL Measured 2026-08-20T05:41:00Z Statistic single; 1 sample Model Qwen3.8 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 118103 Prompt tokens 118103 Generated tokens 32 Prefill 223.2 tok/s Time to first token 14373.6 ms Baseline q38rtm (+1870.5%) Evidence qwen38/results/20260820T054100Z-qwen38-27b-router-cache-verify Benchmark notes. Session B returns with one appended turn. The timed request was a prefix-cache hit: 114895 of 118103 prompt tokens were restored from the host cache and 3208 were processed. One retained sample; an earlier identical replay agreed within 0.2%. Result notes. Time to first token is the server prompt time, 19.71 times lower than the 283232.3 ms cold first visit of the same session. 114895 of 118103 prompt tokens were restored and 3208 processed. Prefill throughput is over the 3208 processed tokens only. An earlier identical replay measured 14396.3 ms, so the two agree within 0.2%. Decode is not recorded: each request generated 32 tokens, too few to measure it.

Qwen3.6

Size: 28B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100	TODO	128	—	—	—	—	305.1	31.8	NA	59%
`q36v` Legacy configuration for qwen36-v100-gpu-services-unloaded1 result · 2 techniques · 0 parameters Configuration ID qwen36-v100-gpu-services-unloaded Created 2026-07-23T02:25:15Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations GPU layer offload All 64 target layers were on the V100. Release GPU capacity Bonsai and GPU vsearch were stopped before the full-offload benchmark. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 64 target layers; 64 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `qwen36-v100-gpu-services-unloaded` Task text generation Status VALID Measured 2026-07-23T02:25:15Z Statistic median; 3 samples Model Qwen3.6 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Prompt tokens 128 Generated tokens 32 Prefill 305.1 tok/s Decode 31.8 tok/s Baseline q36b (+10.7%) Evidence qwen36/README.md Benchmark notes. Bonsai and GPU vsearch unloaded; zero-depth row from the context sweep. Result notes. Zero-depth medians from three samples; Bonsai and GPU vsearch were unloaded.
	llama.cpp/poolside	V100(2)	131072	128	32768	—	—	—	225.9	27.5	NA	—
	llama.cpp/poolside	V100(2)	131072	128	65536	—	—	—	190.1	23.3	NA	—
	llama.cpp/poolside	V100(2)	131072	128	131072	—	—	—	139.0	17.6	NA	—
`q36long` Qwen3.6 27B dual-V100 native long-context depth sweep.3 results · 0 techniques · 2 parameters Configuration ID qwen36-llama04b2b72-2v100-layer-1-1-mg1-b1024-c128k-f16-depth Created 2026-07-27T04:26:17Z Notes Complete GPU placement; 128K fitted context; F16 KV. Applied optimizations No applied technique is cataloged. Uncommon parameters target.fit_min_ctx 131072 tokens target.tensor_split 1/1 — Equal layer split with CUDA1 as the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA0 half of the complete layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA1 half of the complete layer split and main GPU. Results and evidence `qwen36-2v100-depth131072-long-20260727` Task text generation Status VALID Measured 2026-07-27T04:32:03Z Statistic median; 3 samples Model Qwen3.6 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 131072 Prompt tokens 128 Generated tokens 32 Prefill 139.0 tok/s Decode 17.6 tok/s Evidence results/20260727-long-context/qwen36-depth-32k-64k-128k-f16.txt Benchmark notes. Native-context control with complete dual-GPU placement. Result notes. Three-sample medians; decode samples 17.412, 17.605, 17.593. `qwen36-2v100-depth65536-long-20260727` Task text generation Status VALID Measured 2026-07-27T04:28:35Z Statistic median; 3 samples Model Qwen3.6 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 65536 Prompt tokens 128 Generated tokens 32 Prefill 190.1 tok/s Decode 23.3 tok/s Evidence results/20260727-long-context/qwen36-depth-32k-64k-128k-f16.txt Benchmark notes. Native-context control with complete dual-GPU placement. Result notes. Three-sample medians; decode samples 23.178, 23.256, 23.272. `qwen36-2v100-depth32768-long-20260727` Task text generation Status VALID Measured 2026-07-27T04:26:17Z Statistic median; 3 samples Model Qwen3.6 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 32768 Prompt tokens 128 Generated tokens 32 Prefill 225.9 tok/s Decode 27.5 tok/s Evidence results/20260727-long-context/qwen36-depth-32k-64k-128k-f16.txt Benchmark notes. Native-context control with complete dual-GPU placement. Result notes. Three-sample medians; decode samples 26.136, 27.505, 27.570.

Qwen3.6

Size: 28B
Quant: UD-Q4_K_XL

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp/laguna	V100	4096	19	—	—	—	—	119.1	25.9	NA	52%
`q36mb` Legacy configuration for qwen36-v100-mtp-baseline1 result · 0 techniques · 0 parameters Configuration ID qwen36-v100-mtp-baseline Created 2026-07-24T23:20:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `qwen36-v100-mtp-baseline` Task text generation Status VALID Measured 2026-07-24T23:20:00Z Statistic median; 6 samples Model Qwen3.6 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 19 Generated tokens 128 Prefill 119.1 tok/s Decode 25.9 tok/s Evidence qwen36/benchmark-mtp-v100.txt Benchmark notes. Matched native-MTP baseline; GPU services and ComfyUI unloaded. Result notes. Six-sample matched median for the UD-Q4_K_XL MTP artifact.
	llama.cpp/laguna	V100	4096	19	—	Local GPU MTP	+52.2%	89.6%	84.3	39.4	NA	NA
`q36mw` Legacy configuration for qwen36-v100-mtp-w11 result · 4 techniques · 0 parameters Configuration ID qwen36-v100-mtp-w1 Created 2026-07-24T23:25:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Draft placement The native MTP head and target were fully resident on the V100. Speculative decoding The MTP head proposed one token per target verification step. Speculative policy tuning The retained native MTP result used one proposed token. Weight quantization The target artifact used UD-Q4_K_XL weights with its native MTP head. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `qwen36-v100-mtp-w1` Task text generation Status VALID Measured 2026-07-24T23:25:00Z Statistic median; 6 samples Model Qwen3.6 27B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 19 Generated tokens 128 Prefill 84.3 tok/s Decode 39.4 tok/s Baseline q36mb (+52.2%) Output matches baseline yes Evidence qwen36/benchmark-mtp-v100.txt Benchmark notes. Native embedded MTP head with full V100 offload. Result notes. Six-sample width-1 median; all output token sequences matched.

Muse Glimmer 30B

Size: 30B
Quant: K-Quant-17GB

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100	4096	11	—	✕	—	—	82.6	37.8	NA	70%
`mg17bs` Muse Glimmer 17GB greedy server baseline on one V100.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-server-b512-c4k-f16-fa Created 2026-08-10T21:58:01Z Notes Matched control for the DFlash width sweep. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Full offload on physical CUDA0. Results and evidence `muse-glimmer-17gb-server-baseline-20260811` Task text generation Status VALID Measured 2026-08-11T17:00:03Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 128 Prefill 82.6 tok/s Decode 37.8 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Greedy matched control. started_at is the first server journal time. Result notes. Five deterministic target-only samples; raw responses are retained.
	llama.cpp	V100	4096	512	—	✕	—	—	845.2	38.7	40%	72%
`mg17tp` Muse Glimmer 17GB llama-bench on one V100 32GB.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-full-b512-c4k-f16-fa Created 2026-08-10T21:58:01Z Notes Measured target-only throughput point. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Full offload on physical CUDA0. Results and evidence `muse-glimmer-17gb-throughput-20260811` Task text generation Status VALID Measured 2026-08-11T16:55:13Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 512 Generated tokens 128 Prefill 845.2 tok/s Decode 38.7 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/17gb-throughput.jsonl Benchmark notes. started_at is the first retained JSON record timestamp, emitted at the end of its prompt aggregate because exact start was not logged. Result notes. Prefill samples: 788.628, 846.064, 845.697, 845.153, 845.209. Decode samples: 38.5577, 38.7902, 38.7663, 38.6759, 38.6591.
	llama.cpp	V100	32768	128	—	✕	—	—	296.3	38.7	NA	72%
`mgc0` Muse Glimmer 17GB zero-depth control on one V100.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-depth0-b512-c32k-f16-fa Created 2026-08-11T16:57:19Z Notes Maximum tested context depth was 32,768. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target on physical CUDA0. Results and evidence `muse-glimmer-17gb-depth0-20260811` Task text generation Status VALID Measured 2026-08-11T16:57:19Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 0 Prompt tokens 128 Generated tokens 32 Prefill 296.3 tok/s Decode 38.7 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/17gb-context-depth.jsonl Benchmark notes. Zero-depth control. Result notes. Five samples per phase in JSONL.
	llama.cpp	V100	4096	512	—	?	—	—	824.5	—	39%	—
`mg17bad` Invalid Muse Glimmer prefill with a router GPU reload.1 result · 0 techniques · 2 parameters Configuration ID muse-glimmer-17gb-llama030-v100-full-b512-c4k-router-reload Created 2026-08-11T14:49:54Z Notes The model router reloaded on both V100s during generation. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_service_state local-ai-router reloaded on both V100s during generation — This contamination makes the retained prompt aggregate invalid. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Muse target on CUDA0; an unrelated router model reloaded on both V100s during the incomplete generation phase. Results and evidence `muse-glimmer-17gb-prefill-invalid-router-reload-20260811` Task text generation Status INVALID Measured 2026-08-11T14:49:54Z Statistic mean; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 824.5 tok/s Evidence muse-glimmer/results/20260811T144634Z-muse-glimmer-30b-r720/17gb-throughput.jsonl Benchmark notes. Invalid because local-ai-router reloaded an unrelated model on both V100s during generation. No generation aggregate completed. Result notes. Prefill mean from five samples. This value is retained only as invalid evidence and must not be used for comparison.
	llama.cpp	V100	32768	128	2048	✕	—	—	294.6	38.2	NA	71%
`mgc2k` Muse Glimmer 17GB at context depth 2,048 on one V100.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-depth2048-b512-c32k-f16-fa Created 2026-08-11T16:57:47Z Notes Depth cache was built before each timed phase. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target on physical CUDA0. Results and evidence `muse-glimmer-17gb-depth2048-20260811` Task text generation Status VALID Measured 2026-08-11T16:57:47Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 2048 Prompt tokens 128 Generated tokens 32 Prefill 294.6 tok/s Decode 38.2 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/17gb-context-depth.jsonl Benchmark notes. Timed after a 2,048-token depth cache. Result notes. Five samples per phase in JSONL.
	llama.cpp	V100	32768	128	8192	✕	—	—	287.2	38.0	NA	—
`mgc8k` Muse Glimmer 17GB at context depth 8,192 on one V100.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-depth8192-b512-c32k-f16-fa Created 2026-08-11T16:58:18Z Notes Depth cache was built before each timed phase. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target on physical CUDA0. Results and evidence `muse-glimmer-17gb-depth8192-20260811` Task text generation Status VALID Measured 2026-08-11T16:58:18Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 8192 Prompt tokens 128 Generated tokens 32 Prefill 287.2 tok/s Decode 38.0 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/17gb-context-depth.jsonl Benchmark notes. Timed after an 8,192-token depth cache. Result notes. Five samples per phase in JSONL.
	llama.cpp	V100	32768	128	32768	✕	—	—	273.4	36.8	NA	—
`mgc32k` Muse Glimmer 17GB at context depth 32,768 on one V100.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-depth32768-b512-c32k-f16-fa Created 2026-08-11T16:58:58Z Notes Depth cache was built before each timed phase. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target on physical CUDA0. Results and evidence `muse-glimmer-17gb-depth32768-20260811` Task text generation Status VALID Measured 2026-08-11T16:58:58Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 32768 Prompt tokens 128 Generated tokens 32 Prefill 273.4 tok/s Decode 36.8 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/17gb-context-depth.jsonl Benchmark notes. Timed after a 32,768-token depth cache. Result notes. Five samples per phase in JSONL.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	-20.8%	15.0%	79.6	30.0	NA	NA
`mg17d16` Muse Glimmer 17GB with width-16 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-dflash-w16-b512-c4k-f16-fa Created 2026-08-10T21:58:01Z Notes Measured matched speculative point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Target and DFlash placed on physical CUDA0. Results and evidence `muse-glimmer-17gb-dflash-w16-20260811` Task text generation Status PROVISIONAL Measured 2026-08-11T17:03:17Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 128 Prefill 79.6 tok/s Decode 30.0 tok/s Baseline mg17bs (-20.8%) Output matches baseline no Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Width 16 lost to the matched baseline and diverged at generated token 60. started_at is the first server journal time. Result notes. All five samples diverged first at one-based generated token 60.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	+20.1%	78.9%	79.3	45.4	NA	NA
`mg17d1` Muse Glimmer 17GB with width-1 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-dflash-w1-b512-c4k-f16-fa Created 2026-08-11T17:00:40Z Notes Matched greedy server point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-17gb-dflash-w1-20260811` Task text generation Status VALID Measured 2026-08-11T17:00:40Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 79.3 tok/s Decode 45.4 tok/s Baseline mg17bs (+20.1%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Exact greedy output matched the target-only baseline. Result notes. Width one was the fastest exact-output 17 GB point.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	+19.7%	64.0%	79.7	45.3	NA	NA
`mg17d2` Muse Glimmer 17GB with width-2 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-dflash-w2-b512-c4k-f16-fa Created 2026-08-11T17:01:18Z Notes Matched greedy server point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-17gb-dflash-w2-20260811` Task text generation Status VALID Measured 2026-08-11T17:01:18Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 79.7 tok/s Decode 45.3 tok/s Baseline mg17bs (+19.7%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Exact greedy output matched the target-only baseline. Result notes. Width two kept exact output but was slightly slower than width one.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	-1.3%	41.8%	78.9	37.3	NA	NA
`mg17d4` Muse Glimmer 17GB with width-4 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-dflash-w4-b512-c4k-f16-fa Created 2026-08-11T17:01:54Z Notes Output diverged at generated token 45. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-17gb-dflash-w4-20260811` Task text generation Status PROVISIONAL Measured 2026-08-11T17:01:54Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 78.9 tok/s Decode 37.3 tok/s Baseline mg17bs (-1.3%) Output matches baseline no Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Provisional because output diverged at generated token 45. Result notes. All five samples diverged first at one-based generated token 45.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	-16.5%	25.5%	80.0	31.6	NA	NA
`mg17d8` Muse Glimmer 17GB with width-8 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-17gb-llama030-v100-dflash-w8-b512-c4k-f16-fa Created 2026-08-11T17:02:33Z Notes Output diverged at generated token 60. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-17gb-dflash-w8-20260811` Task text generation Status PROVISIONAL Measured 2026-08-11T17:02:33Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 80.0 tok/s Decode 31.6 tok/s Baseline mg17bs (-16.5%) Output matches baseline no Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Provisional because output diverged at generated token 60. Result notes. All five samples diverged first at one-based generated token 60.

Muse Glimmer 30B

Size: 30B
Quant: K-Quant-Dynamic

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100	4096	11	—	✕	—	—	79.6	32.1	NA	70%
`mgdybs` Muse Glimmer dynamic greedy server baseline on one V100.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-server-b512-c4k-f16-fa Created 2026-08-10T21:58:01Z Notes Matched control for the DFlash width sweep. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Full offload on physical CUDA0. Results and evidence `muse-glimmer-dynamic-server-baseline-20260811` Task text generation Status VALID Measured 2026-08-11T17:04:03Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 128 Prefill 79.6 tok/s Decode 32.1 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Greedy matched control. started_at is the first server journal time. Result notes. Five deterministic target-only samples; raw responses are retained.
	llama.cpp	V100	4096	512	—	✕	—	—	897.5	33.0	43%	72%
`mgdytp` Muse Glimmer dynamic llama-bench on one V100 32GB.1 result · 2 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-full-b512-c4k-f16-fa Created 2026-08-10T21:58:01Z Notes Measured target-only throughput point. Applied optimizations Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Full offload on physical CUDA0. Results and evidence `muse-glimmer-dynamic-throughput-20260811` Task text generation Status VALID Measured 2026-08-11T16:56:33Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 512 Generated tokens 128 Prefill 897.5 tok/s Decode 33.0 tok/s Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/dynamic-throughput.jsonl Benchmark notes. started_at is the first retained JSON record timestamp, emitted at the end of its prompt aggregate because exact start was not logged. Result notes. Prefill samples: 798.557, 900.187, 897.461, 898.808, 897.194. Decode samples: 33.1186, 33.0548, 32.9780, 32.8771, 32.7901.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	-17.6%	12.8%	75.8	26.4	NA	NA
`mgdyd16` Muse Glimmer dynamic with width-16 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-dflash-w16-b512-c4k-f16-fa Created 2026-08-10T21:58:01Z Notes Measured matched speculative point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Target and DFlash placed on physical CUDA0. Results and evidence `muse-glimmer-dynamic-dflash-w16-20260811` Task text generation Status VALID Measured 2026-08-11T17:07:26Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 128 Prefill 75.8 tok/s Decode 26.4 tok/s Baseline mgdybs (-17.6%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Width 16 kept exact output but lost to the matched baseline. started_at is the first server journal time. Result notes. Width sixteen kept exact output but lost to baseline.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	+28.1%	82.6%	74.8	41.1	NA	NA
`mgdyd1` Muse Glimmer dynamic with width-1 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-dflash-w1-b512-c4k-f16-fa Created 2026-08-11T17:04:44Z Notes Matched greedy server point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-dynamic-dflash-w1-20260811` Task text generation Status VALID Measured 2026-08-11T17:04:44Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 74.8 tok/s Decode 41.1 tok/s Baseline mgdybs (+28.1%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Exact greedy output matched the target-only baseline. Result notes. Width one improved the dynamic target with exact output.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	+34.8%	66.1%	73.8	43.3	NA	NA
`mgdyd2` Muse Glimmer dynamic with width-2 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-dflash-w2-b512-c4k-f16-fa Created 2026-08-11T17:05:23Z Notes Matched greedy server point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-dynamic-dflash-w2-20260811` Task text generation Status VALID Measured 2026-08-11T17:05:23Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 73.8 tok/s Decode 43.3 tok/s Baseline mgdybs (+34.8%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Exact greedy output matched the target-only baseline. Result notes. Width two was the fastest dynamic point and kept exact output.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	+16.2%	43.5%	78.5	37.3	NA	NA
`mgdyd4` Muse Glimmer dynamic with width-4 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-dflash-w4-b512-c4k-f16-fa Created 2026-08-11T17:06:00Z Notes Matched greedy server point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-dynamic-dflash-w4-20260811` Task text generation Status VALID Measured 2026-08-11T17:06:00Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 78.5 tok/s Decode 37.3 tok/s Baseline mgdybs (+16.2%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Exact greedy output matched the target-only baseline. Result notes. Width four remained faster than baseline and kept exact output.
	llama.cpp	V100	4096	11	—	Local GPU DFlash	-7.2%	25.4%	77.7	29.8	NA	NA
`mgdyd8` Muse Glimmer dynamic with width-8 DFlash on one V100.1 result · 4 techniques · 5 parameters Configuration ID muse-glimmer-dynamic-llama030-v100-dflash-w8-b512-c4k-f16-fa Created 2026-08-11T17:06:41Z Notes Matched greedy server point. Applied optimizations Draft placement The complete DFlash draft was placed beside the target on CUDA0. Flash attention Flash Attention was enabled with F16 K and V cache types. GPU layer offload The complete Muse Glimmer target was placed on physical CUDA0. Speculative decoding The matching DFlash model proposed tokens for target verification. Uncommon parameters node:r720.numa_node0_free_before 10667 MiB MiB — Recorded after checksum verification. node:r720.numa_node1_free_before 3782 MiB MiB — Recorded after checksum verification. system.automatic_numa_balancing enabled — /proc/sys/kernel/numa_balancing was 1. system.gpu_service_state lab-gpu-vram-eaters.slice runtime-masked; services stopped — The mask was removed and services were restored after the run. target.model_revision 93769bc7ab5ad1e9cd22d857e3138cf5d977ae81 — Pinned official Meta GGUF repository revision. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Complete target and DFlash draft on physical CUDA0. Results and evidence `muse-glimmer-dynamic-dflash-w8-20260811` Task text generation Status VALID Measured 2026-08-11T17:06:41Z Statistic median; 5 samples Model Muse Glimmer 30B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 77.7 tok/s Decode 29.8 tok/s Baseline mgdybs (-7.2%) Output matches baseline yes Evidence muse-glimmer/results/20260811T165222Z-muse-glimmer-30b-r720/ Benchmark notes. Exact greedy output matched the target-only baseline. Result notes. Width eight kept exact output but lost to baseline.

Gemma 4

Size: 31B
Quant: UD-Q4_K_XL

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	CPU	TODO	128	—	—	—	—	4.9	2.6	NA	NA
`g31c` Legacy configuration for gemma31-cpu-t161 result · 1 technique · 0 parameters Configuration ID gemma31-cpu-t16 Created 2026-07-19T20:19:31Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations CPU thread tuning The best recorded Gemma 4 31B CPU point used 16 inference threads. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 0 GPU layers; 16 threads; NUMA distribute Results and evidence `gemma31-cpu-t16` Task text generation Status VALID Measured 2026-07-19T20:19:31Z Statistic single; 1 sample Model Gemma 4 31B IT Backend llama.cpp Hosts forge Hardware CPU Prompt tokens 128 Generated tokens 32 Prefill 4.9 tok/s Decode 2.6 tok/s Evidence gemma-xeon/benchmark-gemma-31b.txt Benchmark notes. llama-bench thread sweep. Start time estimated from the retained report mtime, confirmed by its Git import commit. Result notes. pp128 and tg32 from llama-bench.
	llama.cpp	CPU	4096	TODO	—	—	—	—	—	2.5	NA	NA
`g31mb` Legacy configuration for gemma31-mtp-baseline1 result · 0 techniques · 0 parameters Configuration ID gemma31-mtp-baseline Created 2026-07-19T20:19:31Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 0 GPU layers; 16 threads; NUMA distribute Results and evidence `gemma31-mtp-baseline` Task text generation Status VALID Measured 2026-07-19T20:19:31Z Statistic single; 1 sample Model Gemma 4 31B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Generated tokens 128 Decode 2.5 tok/s Evidence gemma-xeon/benchmark-gemma-31b.txt Benchmark notes. Non-speculative baseline from the final MTP confirmation. Start time estimated from the retained report mtime, confirmed by its Git import commit. Result notes. Matched non-speculative baseline from the final MTP confirmation.
	llama.cpp	CPU	4096	TODO	—	Local CPU MTP	+35.4%	77.5%	—	3.4	NA	NA
`g31mw` Legacy configuration for gemma31-mtp-w11 result · 1 technique · 0 parameters Configuration ID gemma31-mtp-w1 Created 2026-07-19T20:19:31Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding The MTP head proposed one token per target verification step. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target_and_draft; CPU; 0 GPU layers; 16 threads; NUMA distribute Results and evidence `gemma31-mtp-w1` Task text generation Status PROVISIONAL Measured 2026-07-19T20:19:31Z Statistic single; 1 sample Model Gemma 4 31B IT Backend llama.cpp Hosts forge Hardware CPU Context window 4096 Generated tokens 128 Prefill 4.9 tok/s Decode 3.4 tok/s Baseline g31mb (+35.4%) Output matches baseline no Evidence gemma-xeon/benchmark-gemma-31b.txt Benchmark notes. Final tuned confirmation. Output diverged at token 117. Start time estimated from the retained report mtime, confirmed by its Git import commit. Result notes. Prefill is the matching pp128 target baseline. The raw speculative prompt timing was not retained. The decode baseline in the confirmation was 2.489 tok/s.
	llama.cpp/laguna	V100	TODO	128	—	—	—	—	258.9	35.6	NA	68%
`g31v` Legacy configuration for gemma31-v1001 result · 2 techniques · 0 parameters Configuration ID gemma31-v100 Created 2026-07-25T00:09:26Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations GPU layer offload The complete dense Gemma 4 31B target fit on the V100 with a 1 GiB margin. Weight quantization The target artifact used UD-Q4_K_XL weights. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `gemma31-v100` Task text generation Status VALID Measured 2026-07-25T00:09:26Z Statistic mean; 6 samples Model Gemma 4 31B IT Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Prompt tokens 128 Generated tokens 32 Prefill 258.9 tok/s Decode 35.6 tok/s Evidence gemma-xeon/benchmark-gemma-31b-v100.jsonl Benchmark notes. Full V100 offload with a 1 GiB fit margin. Result notes. Means of six pp128 and tg32 samples with full V100 offload.
	llama.cpp/poolside	V100	4096	20	—	—	—	—	144.2	33.0	NA	63%
`g31vb` Legacy configuration for gemma31-v100-mtp-baseline1 result · 2 techniques · 0 parameters Configuration ID gemma31-v100-mtp-baseline Created 2026-07-25T00:14:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Flash attention The matched Gemma 4 31B V100 control used flash attention. KV-cache quantization The matched V100 control used Q8_0 K and V caches. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `gemma31-v100-mtp-baseline` Task text generation Status VALID Measured 2026-07-25T00:14:00Z Statistic median; 6 samples Model Gemma 4 31B IT Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 20 Generated tokens 128 Prefill 144.2 tok/s Decode 33.0 tok/s Evidence gemma-xeon/benchmark-gemma-31b-mtp-v100.txt Benchmark notes. Matched Poolside-build baseline with full V100 offload. Result notes. Median of six matched server requests.
	llama.cpp/poolside	CPU + V100	4096	20	—	Local CPU MTP	-4.5%	63.6%	139.1	31.5	NA	NA
`g31vw` Legacy configuration for gemma31-v100-mtp-w11 result · 3 techniques · 0 parameters Configuration ID gemma31-v100-mtp-w1 Created 2026-07-25T00:17:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Flash attention The Gemma 4 31B MTP benchmark used flash attention. KV-cache quantization The MTP benchmark used Q8_0 K and V caches. Speculative decoding The MTP head proposed one token per target verification step. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA distribute r720 — draft; CPU; 0 GPU layers; 16 threads; 16 batch threads; NUMA distribute Results and evidence `gemma31-v100-mtp-w1` Task text generation Status PROVISIONAL Measured 2026-07-25T00:17:00Z Statistic median; 6 samples Model Gemma 4 31B IT Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 20 Generated tokens 128 Prefill 139.1 tok/s Decode 31.5 tok/s Baseline g31vb (-4.5%) Output matches baseline no Evidence gemma-xeon/benchmark-gemma-31b-mtp-v100.txt Benchmark notes. Local CPU MTP assistant; output diverged from the baseline. Result notes. Median of six requests; each had 49/77 accepted draft tokens.

Laguna XS 2.1

Size: 33B / 3B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp/poolside	V100	4096	11	—	✕	—	—	180.1	77.3	NA	NA
`lxsb` Laguna XS 2.1 target-only CUDA0 server control.1 result · 1 technique · 1 parameter Configuration ID laguna-xs-poolside04b2b72-v100-server-c4k-f16-fa Created 2026-07-29T23:07:02Z Notes Matched control for the width-1 DFlash server. Applied optimizations GPU layer offload The complete target was placed on CUDA0. Uncommon parameters system.gpu_services unloaded — The model router and ComfyUI stayed active but released GPU memory. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 40 target layers; 40 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; Complete target placement with room reserved for the matched draft. Results and evidence `laguna-xs-dflash-baseline-20260729` Task text generation Status VALID Measured 2026-07-29T23:07:02Z Statistic median; 5 samples Model Laguna XS 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 32 Prefill 180.1 tok/s Decode 77.3 tok/s Evidence results/20260729T231000Z-remaining-models/baseline-server.log Benchmark notes. Matched target-only server control. Decode varied from 50.419 to 89.489 tok/s; a post-DFlash control was also variable. Result notes. Decode samples: 74.668, 77.281, 82.321, 89.489, 50.419 tok/s.
	llama.cpp/poolside	V100	4096	128	—	✕	—	—	422.0	95.9	NA	NA
`lxsv` Laguna XS 2.1 complete CUDA0 placement.1 result · 2 techniques · 1 parameter Configuration ID laguna-xs-poolside04b2b72-v100-full-b2048-c4k-f16-fa Created 2026-07-29T23:06:44Z Notes Five llama-bench repetitions with a 1 GiB fit margin. Applied optimizations Flash attention Flash Attention was enabled with F16 KV. GPU layer offload The complete Laguna XS target was placed on CUDA0. Uncommon parameters system.gpu_services unloaded — The model router and ComfyUI stayed active but released GPU memory. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 40 target layers; 40 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; CUDA_VISIBLE_DEVICES=0; complete target placement. Results and evidence `laguna-xs-v100-20260729` Task text generation Status VALID Measured 2026-07-29T23:06:44Z Statistic median; 5 samples Model Laguna XS 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 128 Generated tokens 32 Prefill 422.0 tok/s Decode 95.9 tok/s Evidence results/20260729T231000Z-remaining-models/laguna-xs-v100.jsonl Benchmark notes. Complete CUDA0 placement. Five pp128 and five tg32 samples. Result notes. Prefill samples: 422.039, 420.833, 424.766, 418.920, 425.649. Decode samples: 95.8583, 95.8360, 95.9103, 96.0339, 95.5315.
	llama.cpp/poolside	V100	4096	11	—	Local GPU DFlash	+35.3%	50.0%	275.6	104.6	NA	NA
`lxsd` Laguna XS 2.1 with a width-1 DFlash assistant on CUDA0.1 result · 4 techniques · 2 parameters Configuration ID laguna-xs-poolside04b2b72-v100-dflash-w1-c4k-f16-fa Created 2026-07-29T23:07:14Z Notes Target and converted BF16 draft were fully offloaded. Applied optimizations Draft placement The five-layer BF16 DFlash assistant was fully on CUDA0. Flash attention Flash Attention was enabled with F16 KV. GPU layer offload The complete target and draft were placed on CUDA0. Speculative decoding Width one gave 50% token acceptance and a 35.3% matched median gain. Uncommon parameters draft.source_format Safetensors BF16 — Converted with Poolside convert_hf_to_gguf.py at 04b2b72cb540. system.gpu_services unloaded — The model router and ComfyUI stayed active but released GPU memory. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 40 target layers; 40 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; Complete target and five-layer DFlash placement on CUDA0. Results and evidence `laguna-xs-dflash-w1-20260729` Task text generation Status PROVISIONAL Measured 2026-07-29T23:07:14Z Statistic median; 5 samples Model Laguna XS 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 32 Prefill 275.6 tok/s Decode 104.6 tok/s Baseline lxsb (+35.3%) Output matches baseline no Evidence results/20260729T231000Z-remaining-models/dflash-server.log Benchmark notes. The draft was converted from Poolside BF16 Safetensors. Throughput was stable after the first request, but greedy output diverged. Result notes. Decode samples: 50.868, 104.311, 104.638, 104.701, 104.556 tok/s. The width-1 draft was 9.1% faster than the stable 95.858 tok/s llama-bench target control.

Qwen3.6 35B-A3B

Size: 35B / 3B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp/poolside	V100(2)	1024	512	—	—	—	—	486.5	80.3	2%	NA
`q365d` Qwen3.6 35B-A3B equal two-V100 layer split.1 result · 3 techniques · 4 parameters Configuration ID qwen36-35b-poolside04b2b72-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-28T20:32:44Z Notes Complete placement; equal tensor split; CUDA1 main GPU. Applied optimizations GPU link topology The V100s communicate through the PCIe SYS path. GPU peer access NVIDIA peer access is unavailable on this topology. Same-host layer split The split added only 1.4% decode throughput because one V100 already holds the complete model. Uncommon parameters service.pi_model_id qwen3.6-35b-a3b — Pi and the model router use this identifier for the retained profile. system.gpu_services stopped — The model router and ComfyUI were not resident. target.fit_target 1024 MiB — Per-device free-memory margin. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the equal layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the equal layer split and main GPU. Results and evidence `qwen36-35b-2v100-layer-20260728` Task text generation Status VALID Measured 2026-07-28T20:32:44Z Statistic median; 5 samples Model Qwen3.6 35B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 486.5 tok/s Decode 80.3 tok/s Baseline q365s (+1.4%) Output matches baseline yes Evidence results/20260728-qwen36-35b-a3b-v100/dual-v100.txt Benchmark notes. Complete equal layer split. One generation sample was a low outlier. Result notes. Decode was 1.4% faster. Fixed-seed greedy output matched byte for byte; output SHA-256 38b138cc085b2ffb5a05e38f224e2f8b5b49db59b3462e1cc6e7a391d1fdd4fa.
	llama.cpp/poolside	V100	4096	11	—	—	—	—	76.4	74.7	NA	NA
`q365mb` Qwen3.6 35B-A3B one-V100 server baseline for the MTP sweep.1 result · 0 techniques · 1 parameter Configuration ID qwen36-35b-poolside04b2b72-v100-b128-c4k-f16-server Created 2026-07-31T03:36:24Z Notes Complete CUDA0 placement; F16 KV; Flash Attention. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The original two-V100 Qwen service was stopped for the matched sweep. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; CUDA0 held the complete target. Results and evidence `qwen36-35b-mtp-baseline-20260731` Task text generation Status VALID Measured 2026-07-31T03:36:24Z Statistic median; 5 samples Model Qwen3.6 35B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 76.4 tok/s Decode 74.7 tok/s Evidence results/20260731T033557Z-qwen36-35b-mtp-v100/README.md Benchmark notes. Matched non-speculative server baseline with complete CUDA0 offload. Result notes. Decode samples: 74.8068, 74.2760, 74.6514, 74.7114, 63.5899 tok/s.
	llama.cpp/poolside	V100	1024	512	—	—	—	—	488.1	79.2	2%	NA
`q365s` Qwen3.6 35B-A3B complete one-V100 placement.1 result · 0 techniques · 2 parameters Configuration ID qwen36-35b-poolside04b2b72-v100-fit1g-b128-c1k-f16 Created 2026-07-28T20:31:55Z Notes Automatic fit with a 1 GiB margin; F16 KV; GPU services stopped. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.fit_target 1024 MiB — Per-device free-memory margin. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 holds the complete fitted model. Results and evidence `qwen36-35b-v100-fit-20260728` Task text generation Status VALID Measured 2026-07-28T20:31:55Z Statistic median; 5 samples Model Qwen3.6 35B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 488.1 tok/s Decode 79.2 tok/s Evidence results/20260728-qwen36-35b-a3b-v100/single-v100.txt Benchmark notes. Complete CUDA0 placement. The fifth sample was a low outlier. Result notes. Five-sample medians. Prompt samples: 489.672, 484.621, 489.991, 488.104, 382.186. Decode samples: 79.2416, 79.2163, 78.9726, 79.2278, 66.6857.
	llama.cpp/poolside	V100	4096	11	—	Local GPU MTP	+29.8%	84.1%	77.0	96.9	NA	NA
`q365m1` Qwen3.6 35B-A3B one-V100 native MTP with proposal width one.1 result · 3 techniques · 2 parameters Configuration ID qwen36-35b-poolside04b2b72-v100-b128-c4k-f16-mtp-q4-0-w1 Created 2026-07-31T03:36:52Z Notes Complete target and Q4_0 MTP placement on CUDA0. Applied optimizations Draft placement The standalone Q4_0 MTP assistant was fully on CUDA0. Speculative decoding Width one gave 84.1% acceptance and a 29.8% matched median gain. Speculative policy tuning Proposal width one was part of the matched width sweep. Uncommon parameters draft.draft_p_min 0 — Every MTP proposal was eligible for target verification. system.gpu_services stopped — The original two-V100 Qwen service was stopped for the matched sweep. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; CUDA0 held the complete target and MTP assistant. Results and evidence `qwen36-35b-mtp-w1-20260731` Task text generation Status PROVISIONAL Measured 2026-07-31T03:36:52Z Statistic median; 5 samples Model Qwen3.6 35B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 77.0 tok/s Decode 96.9 tok/s Baseline q365mb (+29.8%) Output matches baseline no Evidence results/20260731T033557Z-qwen36-35b-mtp-v100/README.md Benchmark notes. Provisional because the prose output diverged from the baseline. Result notes. Decode samples: 96.7209, 96.1196, 96.8700, 96.8604, 96.9291 tok/s.
	llama.cpp/poolside	V100	4096	11	—	Local GPU MTP	+35.9%	59.6%	76.8	101.5	NA	NA
`q365m2` Qwen3.6 35B-A3B one-V100 native MTP with proposal width two.1 result · 3 techniques · 2 parameters Configuration ID qwen36-35b-poolside04b2b72-v100-b128-c4k-f16-mtp-q4-0-w2 Created 2026-07-31T03:37:20Z Notes Best median throughput in the width sweep. Applied optimizations Draft placement The standalone Q4_0 MTP assistant was fully on CUDA0. Speculative decoding Width two gave 59.6% acceptance and a 35.9% matched median gain. Speculative policy tuning Proposal width two gave the best median decode throughput. Uncommon parameters draft.draft_p_min 0 — Every MTP proposal was eligible for target verification. system.gpu_services stopped — The original two-V100 Qwen service was stopped for the matched sweep. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; CUDA0 held the complete target and MTP assistant. Results and evidence `qwen36-35b-mtp-w2-20260731` Task text generation Status PROVISIONAL Measured 2026-07-31T03:37:20Z Statistic median; 5 samples Model Qwen3.6 35B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 76.8 tok/s Decode 101.5 tok/s Baseline q365mb (+35.9%) Output matches baseline no Evidence results/20260731T033557Z-qwen36-35b-mtp-v100/README.md Benchmark notes. Best tested width. Arithmetic and JSON checks matched the baseline; the prose token sequence diverged. Result notes. Decode samples: 101.4608, 102.6336, 101.4050, 102.9975, 101.0605 tok/s. Arithmetic and JSON task checks matched exactly.
	llama.cpp/poolside	V100	4096	11	—	Local GPU MTP	+25.6%	46.2%	74.9	93.8	NA	NA
`q365m3` Qwen3.6 35B-A3B one-V100 native MTP with proposal width three.1 result · 3 techniques · 2 parameters Configuration ID qwen36-35b-poolside04b2b72-v100-b128-c4k-f16-mtp-q4-0-w3 Created 2026-07-31T03:37:45Z Notes Complete target and Q4_0 MTP placement on CUDA0. Applied optimizations Draft placement The standalone Q4_0 MTP assistant was fully on CUDA0. Speculative decoding Width three gave 46.2% acceptance and a 25.6% matched median gain. Speculative policy tuning Proposal width three reduced acceptance and throughput. Uncommon parameters draft.draft_p_min 0 — Every MTP proposal was eligible for target verification. system.gpu_services stopped — The original two-V100 Qwen service was stopped for the matched sweep. Node placement r720 — target_and_draft; CUDA; Tesla PG500-216 (V100 32GB); 99 GPU layers; 16 threads; 16 batch threads; NUMA interleave; CUDA0 held the complete target and MTP assistant. Results and evidence `qwen36-35b-mtp-w3-20260731` Task text generation Status PROVISIONAL Measured 2026-07-31T03:37:45Z Statistic median; 5 samples Model Qwen3.6 35B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 128 Prefill 74.9 tok/s Decode 93.8 tok/s Baseline q365mb (+25.6%) Output matches baseline no Evidence results/20260731T033557Z-qwen36-35b-mtp-v100/README.md Benchmark notes. Provisional because the prose output diverged from the baseline. Result notes. Decode samples: 93.1148, 93.7596, 93.8018, 93.8952, 87.3935 tok/s.

Dolphin 2.2

Size: 70B
Quant: Q3_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	PipeInfer/lan	CPU + V100	1024	TODO	—	✕	—	—	—	2.4	NA	NA
`pi-2` Legacy configuration for pipeinfer70-v100-target-35-651 result · 1 technique · 0 parameters Configuration ID pipeinfer70-v100-target-35-65 Created 2026-07-22T23:16:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Distributed target inference Forge ran 28 target layers and the R720 ran 52 target layers over 1 GbE. Uncommon parameters No uncommon parameters are cataloged. Node placement forge — target; CPU; 28 target layers; 0 GPU layers; 16 threads; 16 batch threads; NUMA numa r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 52 target layers; 52 GPU layers; 20 threads; 20 batch threads; NUMA numa Results and evidence `pipeinfer70-v100-target-35-65` Task text generation Status VALID Measured 2026-07-22T23:16:00Z Statistic single; 1 sample Model Dolphin 2.2 70B Backend PipeInfer Hosts forge,r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Generated tokens 128 Decode 2.4 tok/s Time to first token 512.0 ms Baseline pi-l (-0.9%) Evidence prisma/V100.md Benchmark notes. Forge owns 28 target layers; R720 owns and offloads 52 layers. Result notes. Two-node 128-token target final. ITL 0.424 s; TTFT 0.512 s.
	PipeInfer/local	CPU + V100	1024	TODO	—	✕	—	—	—	2.4	NA	NA
`pi-l` Legacy configuration for pipeinfer70-v100-ngl521 result · 2 techniques · 0 parameters Configuration ID pipeinfer70-v100-ngl52 Created 2026-07-22T23:16:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations GPU layer offload The V100 held 52 of 80 target layers on the R720. Weight quantization The target artifact used Q3_K_M weights. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 52 GPU layers; 20 threads; 20 batch threads; NUMA numa Results and evidence `pipeinfer70-v100-ngl52` Task text generation Status VALID Measured 2026-07-22T23:16:00Z Statistic single; 1 sample Model Dolphin 2.2 70B Backend PipeInfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Generated tokens 128 Decode 2.4 tok/s Time to first token 417.0 ms Evidence prisma/V100.md Benchmark notes. R720-only final with 52 V100 layers. Result notes. R720-only 128-token final. ITL 0.420 s; TTFT 0.417 s.
	llama.cpp/poolside	V100(2)	32768	128	—	✕	—	—	140.6	5.7	NA	21%
	llama.cpp/poolside	V100	32768	128	8192	✕	—	—	127.9	11.5	NA	—
	llama.cpp/poolside	V100	32768	128	32768	✕	—	—	104.2	10.4	NA	—
`d70ctx` Dolphin 70B dual-V100 context-depth sweep.3 results · 0 techniques · 1 parameter Configuration ID dolphin70-llama04b2b72-2v100-layer-1-1-mg1-b128-c32k-f16-depth Created 2026-07-27T02:35:45Z Notes Equal layer split; CUDA1 main GPU; 32K context allocation. Applied optimizations No applied technique is cataloged. Uncommon parameters target.fit_min_ctx 32768 tokens Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA0 half of the layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA1 half of the layer split and main GPU. Results and evidence `dolphin70-2v100-depth32768-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T02:38:50Z Statistic median; 5 samples Model Dolphin 2.2 70B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 32768 Prompt tokens 128 Generated tokens 32 Prefill 104.2 tok/s Decode 10.4 tok/s Evidence results/20260726-dual-v100/dolphin70-dual-depth-0-8k-32k.txt Benchmark notes. Capacity is demonstrated; decode result remains provisional. Result notes. Decode samples: 10.424, 10.410, 9.664, 10.394, 7.055 tok/s. `dolphin70-2v100-depth8192-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T02:36:43Z Statistic median; 5 samples Model Dolphin 2.2 70B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 8192 Prompt tokens 128 Generated tokens 32 Prefill 127.9 tok/s Decode 11.5 tok/s Evidence results/20260726-dual-v100/dolphin70-dual-depth-0-8k-32k.txt Benchmark notes. Decode result is provisional because the sweep showed two states. Result notes. Decode samples: 11.527, 11.680, 11.632, 11.524, 10.342 tok/s. `dolphin70-2v100-depth0-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T02:35:45Z Statistic median; 5 samples Model Dolphin 2.2 70B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 0 Prompt tokens 128 Generated tokens 32 Prefill 140.6 tok/s Decode 5.7 tok/s Evidence results/20260726-dual-v100/dolphin70-dual-depth-0-8k-32k.txt Benchmark notes. Decode was bimodal across this run and a repeated zero-depth run. Result notes. Repeat decode samples were 12.316, 12.370, 11.882, 5.948, 6.444.
	llama.cpp/poolside	V100(2)	1024	512	—	✕	—	—	138.6	8.4	16%	31%
`d70d` Dolphin 70B split by layer across two V100s.1 result · 3 techniques · 1 parameter Configuration ID dolphin70-llama04b2b72-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-27T01:13:37Z Notes Equal tensor split; CUDA1 main GPU; PCIe SYS path without peer access. Applied optimizations GPU layer offload The second V100 provided more GPU-resident model capacity. GPU link topology Layer split limited transfers across the PCIe SYS path. GPU peer access NVIDIA reported peer reads and writes as topology not supported. Uncommon parameters system.gpu_peer_access unavailable — nvidia-smi topo reports TNS in both directions. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA0 half of the layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA1 half of the layer split and main GPU. Results and evidence `dolphin70-2v100-layer-20260727` Task text generation Status VALID Measured 2026-07-27T01:13:37Z Statistic median; 5 samples Model Dolphin 2.2 70B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 138.6 tok/s Decode 8.4 tok/s Baseline d70s (+31.5%) Output matches baseline yes Evidence results/20260726-dual-v100/dolphin70-dual-layer-1-1-mg1.txt Benchmark notes. Equal layer split with CUDA1 as main GPU. Result notes. Decode +31.5%; prefill +30.0%; byte-identical fixed-seed output.
	llama.cpp/poolside	V100	1024	512	—	✕	—	—	106.6	6.4	12%	24%
`d70s` Dolphin 70B fitted to one V100 with 128-token batches.1 result · 0 techniques · 0 parameters Configuration ID dolphin70-llama04b2b72-v100-fit1g-b128-c1k-f16 Created 2026-07-27T01:10:38Z Notes Matched one-card control for the dual-V100 layer split. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 16 threads; 16 batch threads; NUMA distribute; CUDA0 only; automatic fit with a 1 GiB reserve. Results and evidence `dolphin70-v100-fit-20260727` Task text generation Status VALID Measured 2026-07-27T01:10:38Z Statistic median; 5 samples Model Dolphin 2.2 70B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 106.6 tok/s Decode 6.4 tok/s Evidence results/20260726-dual-v100/dolphin70-single-gpu0.txt Benchmark notes. One-card matched control. Fixed-seed output matched the dual run. Result notes. Five-sample medians; fixed-seed control retained separately.

Qwen2.5

Size: 72B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	prima.cpp	CPU + P40	TODO	TODO	—	✕	—	—	1.5	2.0	NA	NA
`q72p0` Legacy configuration for qwen72-r720-baseline1 result · 0 techniques · 0 parameters Configuration ID qwen72-r720-baseline Created 2026-07-16T20:13:27Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla P40; 80 target layers; 40 GPU layers; 16 threads Results and evidence `qwen72-r720-baseline` Task text generation Status VALID Measured 2026-07-16T20:13:27Z Statistic single; 1 sample Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720 Hardware Tesla P40 Generated tokens 128 Prefill 1.5 tok/s Decode 2.0 tok/s Evidence prisma/POC.md Benchmark notes. 40 target layers on the P40 and 40 on R720 CPUs. Start time is from the retained result directory name. Result notes. Architecture-correct R720 run.
	prima.cpp	CPU + P40	1024	TODO	—	Local CPU 3B draft	+6.1%	66.0%	—	2.1	NA	NA
`q72pl` Legacy configuration for qwen72-local-draft-w11 result · 1 technique · 0 parameters Configuration ID qwen72-local-draft-w1 Created 2026-07-18T02:21:52Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding A separate local draft model proposed one token per target verification step. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target_and_draft; CPU+CUDA; Tesla P40; 80 target layers; 40 GPU layers; 16 threads Results and evidence `qwen72-local-draft-w1` Task text generation Status VALID Measured 2026-07-18T02:21:52Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720 Hardware Tesla P40 Context window 1024 Prefill 1.5 tok/s Decode 2.1 tok/s Baseline q72p0 (+6.1%) Output matches baseline yes Evidence prisma/MTP.md Benchmark notes. Median of three local one-token controls. Start time is the first retained sample directory. Result notes. Prefill is the matching R720 target baseline. The raw speculative prompt timing was not retained. Decode is the median compared with the 1.96 tok/s baseline.
	prima.cpp	CPU + P40	1024	TODO	—	Remote CPU 3B draft	+13.9%	66.0%	—	2.2	NA	NA
`q72pr` Legacy configuration for qwen72-remote-draft-forge-w11 result · 1 technique · 0 parameters Configuration ID qwen72-remote-draft-forge-w1 Created 2026-07-18T02:17:51Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding A remote CPU draft model proposed one token asynchronously. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla P40; 80 target layers; 40 GPU layers; 16 threads forge — draft; CPU; 0 GPU layers; 16 threads; 16 batch threads Results and evidence `qwen72-remote-draft-forge-w1` Task text generation Status VALID Measured 2026-07-18T02:17:51Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720,forge Hardware Tesla P40 Context window 1024 Prefill 1.5 tok/s Decode 2.2 tok/s Baseline q72p0 (+13.9%) Output matches baseline yes Evidence prisma/MTP.md Benchmark notes. Three remote runs: 2.232, 2.191, and 2.235 tok/s. Start time is the first retained sample directory. Result notes. Prefill is the matching R720 target baseline. The raw remote-draft prompt timing was not retained. Remote decode was 7.4% faster than the equivalent local policy.
	prima.cpp	CPU + V100	1024	19	—	✕	—	—	0.3	2.3	NA	NA
`q72v0` Legacy configuration for qwen72-v100-baseline1 result · 0 techniques · 0 parameters Configuration ID qwen72-v100-baseline Created 2026-07-22T22:38:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 40 GPU layers; 16 threads; 16 batch threads Results and evidence `qwen72-v100-baseline` Task text generation Status VALID Measured 2026-07-22T22:38:00Z Statistic single; 1 sample Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 128 Prefill 0.3 tok/s Decode 2.3 tok/s Evidence prisma/V100.md Benchmark notes. 40 target layers on the V100 and 40 on R720 CPUs. Result notes. One 128-token target run with 40 V100 layers.
	prima.cpp	CPU + V100	1024	19	—	✕	—	—	7.5	4.1	NA	NA
`q72i` Qwen2.5 72B target with 60 V100 layers and interleaved page cache.1 result · 3 techniques · 0 parameters Configuration ID qwen72-prima1ea4707-v100-ngl60-b128-c1k-f16-interleave-target Created 2026-07-26T06:10:00Z Notes Retained maximum-throughput Qwen 72B profile; target-only. Applied optimizations GPU layer offload Sixty of 80 layers on the V100, unchanged from the prior profile. Host page cache The page cache was dropped once so the mmap load could place model pages under the interleave policy; the placement persists warm. NUMA memory policy Mapped pages were re-faulted under numactl --interleave=all for a 47.8% decode gain over the one-node page-cache placement. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 60 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Mapped model pages faulted under numactl --interleave=all after a page-cache drop; 60 of 80 layers on the V100. Results and evidence `qwen72-v100-interleave-20260726` Task text generation Status VALID Measured 2026-07-26T06:10:00Z Statistic median; 5 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 128 Prefill 7.5 tok/s Decode 4.1 tok/s Baseline q72o0 (+47.8%) Output matches baseline yes Evidence prisma/QWEN72-OPTIMIZATION-20260725.md Benchmark notes. Target-only prima with the model page cache faulted under numactl --interleave=all. Greedy output byte-identical to the non-interleaved control. Result notes. Medians of five prima-cli runs. 47.8% over the matched 2.76 tok/s control; greedy text byte-identical to the control.
	prima.cpp	CPU + V100	1024	19	—	✕	—	—	7.0	2.8	NA	NA
`q72o0` Qwen2.5 72B target with 60 V100 layers and 128-token batches.1 result · 7 techniques · 9 parameters Configuration ID qwen72-prima1ea4707-v100-ngl60-b128-c1k-f16-target Created 2026-07-25T06:30:00Z Notes Matched target configuration for the optimized remote-draft run. Applied optimizations Architecture-specific build The Prima target used an Ivy Bridge and CUDA compute-70 build. Batch-size tuning Batch and microbatch were both 128 so that 60 target layers fit on the V100. GPU layer offload The V100 held 60 of 80 target layers; the R720 CPUs held 20. Host page cache All target shards were warm in the R720 host page cache. Output tensor placement The Prima backend kept the target output tensor on the V100. Release GPU capacity ComfyUI, the model router, and GPU embeddings were stopped. Runtime selection The benchmark kept the measured optimized Prima target path as its control. Uncommon parameters accelerator:r720-v100-32gb.clock_policy automatic accelerator:r720-v100-32gb.ecc_mode enabled boolean accelerator:r720-v100-32gb.persistence_mode disabled boolean backend.cuda_compute_buffer 425.59 MiB MiB — Reduced from 739.85 MiB by the 128-token microbatch. backend.cuda_kernel_selection automatic — Neither MMQ nor cuBLAS was forced. system.cpu_frequency_governor schedutil system.model_cache_state warm — All target shards were resident in the host page cache. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.output_tensor_placement CUDA — Prima --keep-out-in-cuda. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 60 GPU layers; 16 threads; 16 batch threads Results and evidence `qwen72-v100-ngl60-b128-baseline` Task text generation Status VALID Measured 2026-07-25T06:30:00Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 128 Prefill 7.0 tok/s Decode 2.8 tok/s Evidence prisma/QWEN72-OPTIMIZATION-20260725.md Benchmark notes. Matched control with 60 V100 layers, batch 128, and microbatch 128. Result notes. Median of three matched non-speculative runs.
	llama.cpp/poolside	V100(2)	32768	128	—	✕	—	—	127.5	8.3	NA	—
	llama.cpp/poolside	V100(2)	32768	128	8192	✕	—	—	118.0	8.6	NA	—
	llama.cpp/poolside	V100(2)	32768	128	32768	✕	—	—	95.8	6.6	NA	—
`q72ctx` Qwen2.5 72B dual-V100 context-depth sweep.3 results · 0 techniques · 1 parameter Configuration ID qwen72-llama04b2b72-2v100-layer-1-1-mg1-b128-c32k-f16-depth Created 2026-07-27T02:24:58Z Notes Equal layer split; CUDA1 main GPU; 32K context allocation. Applied optimizations No applied technique is cataloged. Uncommon parameters target.fit_min_ctx 32768 tokens Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA0 half of the layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA1 half of the layer split and main GPU. Results and evidence `qwen72-2v100-depth32768-20260727` Task text generation Status VALID Measured 2026-07-27T02:28:29Z Statistic median; 5 samples Model Qwen2.5 72B Instruct Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 32768 Prompt tokens 128 Generated tokens 32 Prefill 95.8 tok/s Decode 6.6 tok/s Evidence results/20260726-dual-v100/qwen72-dual-depth-0-8k-32k.txt Benchmark notes. Timed after building a 32768-token KV cache. Result notes. Decode retains 79.4% of the zero-depth median. `qwen72-2v100-depth8192-20260727` Task text generation Status VALID Measured 2026-07-27T02:25:58Z Statistic median; 5 samples Model Qwen2.5 72B Instruct Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 8192 Prompt tokens 128 Generated tokens 32 Prefill 118.0 tok/s Decode 8.6 tok/s Evidence results/20260726-dual-v100/qwen72-dual-depth-0-8k-32k.txt Benchmark notes. Timed after building an 8192-token KV cache. Result notes. Decode samples: 7.702, 8.588, 8.683, 8.642, 8.899 tok/s. `qwen72-2v100-depth0-20260727` Task text generation Status VALID Measured 2026-07-27T02:24:58Z Statistic median; 5 samples Model Qwen2.5 72B Instruct Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 0 Prompt tokens 128 Generated tokens 32 Prefill 127.5 tok/s Decode 8.3 tok/s Evidence results/20260726-dual-v100/qwen72-dual-depth-0-8k-32k.txt Benchmark notes. Zero-depth control in the 32K-capacity sweep. Result notes. Decode samples: 9.449, 6.860, 7.132, 8.308, 8.457 tok/s.
	llama.cpp/poolside	V100(2)	1024	512	—	✕	—	—	117.8	8.3	14%	—
`q72d2` Qwen2.5 72B split by layer across two V100s.1 result · 3 techniques · 1 parameter Configuration ID qwen72-llama04b2b72-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-27T01:25:25Z Notes Equal tensor split; CUDA1 main GPU; complete GPU placement. Applied optimizations GPU layer offload Complete GPU placement removed the one-card CPU bottleneck. GPU link topology Layer split limited transfers across the PCIe SYS path. GPU peer access NVIDIA reported peer reads and writes as topology not supported. Uncommon parameters system.gpu_peer_access unavailable — nvidia-smi topo reports TNS in both directions. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA0 half of the complete layer split. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute; CUDA1 half of the complete layer split and main GPU. Results and evidence `qwen72-2v100-layer-20260727` Task text generation Status VALID Measured 2026-07-27T01:25:25Z Statistic median; 5 samples Model Qwen2.5 72B Instruct Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 117.8 tok/s Decode 8.3 tok/s Baseline q72s2 (+218.4%) Output matches baseline yes Evidence results/20260726-dual-v100/qwen72-dual-layer-1-1-mg1.txt Benchmark notes. Complete dual-GPU placement with an equal layer split. Result notes. Decode +218.4%; prefill +331.1%; byte-identical fixed-seed output.
	llama.cpp/poolside	V100(2)	1024	512	—	✕	—	—	27.3	2.6	NA	NA
`q72s2` Qwen2.5 72B fitted across CPU and one V100.1 result · 0 techniques · 0 parameters Configuration ID qwen72-llama04b2b72-v100-fit1g-b128-c1k-f16 Created 2026-07-27T01:18:07Z Notes Matched llama.cpp control for complete two-V100 offload. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 16 threads; 16 batch threads; NUMA distribute; CUDA0 with automatic tensor fit; remaining tensors on CPU. Results and evidence `qwen72-v100-llama-fit-20260727` Task text generation Status VALID Measured 2026-07-27T01:18:07Z Statistic median; 5 samples Model Qwen2.5 72B Instruct Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 27.3 tok/s Decode 2.6 tok/s Evidence results/20260726-dual-v100/qwen72-single-gpu0.txt Benchmark notes. One-card matched llama.cpp control with automatic tensor fit. Result notes. Five-sample medians; one-card CPU and CUDA path.
	prima.cpp	CPU + V100	1024	19	—	Local CPU 3B draft	+11.1%	68.6%	3.6	2.6	NA	NA
`q72vl` Legacy configuration for qwen72-v100-local-draft-w11 result · 1 technique · 0 parameters Configuration ID qwen72-v100-local-draft-w1 Created 2026-07-22T22:46:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding A separate local draft model proposed one token per target verification step. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 40 GPU layers; 16 threads; 16 batch threads Results and evidence `qwen72-v100-local-draft-w1` Task text generation Status VALID Measured 2026-07-22T22:46:00Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 87 Prefill 3.6 tok/s Decode 2.6 tok/s Baseline q72v0 (+11.1%) Evidence prisma/V100.md Benchmark notes. Median of 2.462, 2.600, and 2.609 tok/s. Result notes. Median of three one-token local draft runs.
	prima.cpp	CPU + V100	1024	19	—	Remote CPU 3B draft	-3.1%	66.0%	—	4.0	NA	NA
`q72ir` Qwen2.5 72B interleaved target with one-token forge drafting.1 result · 3 techniques · 0 parameters Configuration ID qwen72-prima1ea4707-v100-ngl60-b128-c1k-f16-interleave-remote-w1 Created 2026-07-26T05:45:00Z Notes Retired: speculation no longer beats the interleaved target. Applied optimizations Draft placement Forge async draft. Acceptance was unchanged at 66%, but the faster target loop removed the end-to-end gain. NUMA memory policy Same interleaved target placement as the target-only profile. Speculative decoding A remote CPU draft model proposed one token asynchronously. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 60 GPU layers; 16 threads; 16 batch threads; NUMA interleave; Same interleaved target placement as the target-only profile. forge — draft; CPU; 0 GPU layers; 16 threads; 16 batch threads; Forge async one-token draft server over 1 GbE. Results and evidence `qwen72-v100-interleave-remote-w1-20260726` Task text generation Status VALID Measured 2026-07-26T05:45:00Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720,forge Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 89 Decode 4.0 tok/s Baseline q72i (-3.1%) Evidence prisma/QWEN72-OPTIMIZATION-20260725.md Benchmark notes. Forge remote draft on the interleaved target. The faster target loop erases the speculative gain; one sample stalled 100 seconds in async network wait. The retained profile is target-only. Result notes. Median of three runs (0.98, 3.95, 4.11). Speculation no longer beats the 4.08 tok/s interleaved target-only profile.
	prima.cpp	CPU + V100	1024	19	—	Remote CPU 3B draft	+39.9%	66.0%	6.6	3.9	NA	NA
`q72o1` Qwen2.5 72B with 60 V100 layers and one-token forge drafting.1 result · 10 techniques · 10 parameters Configuration ID qwen72-prima1ea4707-v100-ngl60-b128-c1k-f16-remote-w1 Created 2026-07-25T06:25:00Z Notes Maximum-throughput profile from the 2026-07-25 optimization. Applied optimizations Architecture-specific build The Prima target used an Ivy Bridge and CUDA compute-70 build. Batch-size tuning Batch and microbatch were both 128 so that 60 target layers fit on the V100. Draft placement The target used the R720 V100 and the asynchronous draft used forge CPUs. GPU layer offload The V100 held 60 of 80 target layers; the R720 CPUs held 20. Host page cache Target and draft shards were warm in their host page caches. Output tensor placement The Prima backend kept the target output tensor on the V100. Release GPU capacity ComfyUI, the model router, and GPU embeddings were stopped. Runtime selection The benchmark used the optimized Prima target and asynchronous draft path. Speculative decoding A remote CPU draft model proposed one token asynchronously. Weight quantization The target and draft artifacts used Q4_K_M weights. Uncommon parameters accelerator:r720-v100-32gb.clock_policy automatic accelerator:r720-v100-32gb.ecc_mode enabled boolean accelerator:r720-v100-32gb.free_vram_at_decode 491 MiB MiB — One-second nvidia-smi sample. accelerator:r720-v100-32gb.persistence_mode disabled boolean backend.cuda_compute_buffer 425.59 MiB MiB — Reduced from 739.85 MiB by the 128-token microbatch. backend.cuda_kernel_selection automatic — Neither MMQ nor cuBLAS was forced. system.cpu_frequency_governor schedutil system.model_cache_state warm — All target and draft shards were in host page caches. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.output_tensor_placement CUDA — Prima --keep-out-in-cuda. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 60 GPU layers; 16 threads; 16 batch threads forge — draft; CPU; 0 GPU layers; 16 threads; 16 batch threads Results and evidence `qwen72-v100-ngl60-b128-remote-w1` Task text generation Status VALID Measured 2026-07-25T06:25:00Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720,forge Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 89 Prefill 6.6 tok/s Decode 3.9 tok/s Baseline q72o0 (+39.9%) Output matches baseline no Evidence prisma/QWEN72-OPTIMIZATION-20260725.md Benchmark notes. Forge CPU draft; 60 V100 target layers; batch and microbatch 128. Result notes. Medians of three runs. Task checks passed; exact paragraph output diverged.
	prima.cpp	CPU + V100	1024	19	—	Remote CPU 3B draft	+14.6%	68.6%	4.0	2.7	NA	NA
`q72vr` Legacy configuration for qwen72-v100-remote-draft-w11 result · 1 technique · 0 parameters Configuration ID qwen72-v100-remote-draft-w1 Created 2026-07-22T22:53:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding A remote CPU draft model proposed one token asynchronously. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 80 target layers; 40 GPU layers; 16 threads; 16 batch threads forge — draft; CPU; 0 GPU layers; 16 threads; 16 batch threads Results and evidence `qwen72-v100-remote-draft-w1` Task text generation Status VALID Measured 2026-07-22T22:53:00Z Statistic median; 3 samples Model Qwen2.5 72B Instruct Backend prima.cpp Hosts r720,forge Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Prompt tokens 19 Generated tokens 86 Prefill 4.0 tok/s Decode 2.7 tok/s Baseline q72v0 (+14.6%) Evidence prisma/V100.md Benchmark notes. Median of 2.681, 1.984, and 2.752 tok/s. Result notes. Median of three asynchronous forge draft runs.

Qwen3-Coder-Next 80B-A3B

Size: 80B / 3B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(2)	1024	512	—	✕	—	—	270.5	79.6	1%	NA
`q3cd` Qwen3-Coder-Next split by layer across two V100s.1 result · 4 techniques · 2 parameters Configuration ID qwen3coder-llamab64739e-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-27T20:03:32Z Notes Equal tensor split; CUDA1 main GPU; complete GPU placement. Applied optimizations GPU layer offload The second card removed the one-card host-weight bottleneck. GPU link topology Layer split limited traffic across the PCIe SYS path. GPU peer access NVIDIA peer access was unavailable on this topology. Same-host layer split Complete model placement across two V100s. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `qwen3coder-2v100-layer-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T20:03:32Z Statistic median; 5 samples Model Qwen3-Coder-Next 80B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 270.5 tok/s Decode 79.6 tok/s Baseline q3cs (+178.0%) Evidence results/20260727-new-model-dual-v100/qwen3-coder-next-dual-layer-1-1-mg1.txt Benchmark notes. Complete GPU placement; correctness check pending. Result notes. Decode 2.78 times the one-card control.
	llama.cpp	CPU + V100	1024	512	—	✕	—	—	52.2	28.6	NA	NA
`q3cs` Qwen3-Coder-Next fitted to one V100.1 result · 0 techniques · 1 parameter Configuration ID qwen3coder-llamab64739e-v100-fit1g-b128-c1k-f16 Created 2026-07-27T20:14:24Z Notes CUDA0 with automatic CPU fallback. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 with automatic fit; remaining tensors on CPU. Results and evidence `qwen3coder-v100-fit-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T20:14:24Z Statistic median; 5 samples Model Qwen3-Coder-Next 80B-A3B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 52.2 tok/s Decode 28.6 tok/s Evidence results/20260727-new-model-dual-v100/qwen3-coder-next-single-gpu0.txt Benchmark notes. One-card control; correctness check pending. Result notes. Five-sample medians.

gpt-oss-120B

Size: 117B / 5B
Quant: MXFP4

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(2)	1024	512	—	✕	—	—	627.4	99.9	5%	NA
`god` gpt-oss-120B split by layer across two V100s.1 result · 4 techniques · 2 parameters Configuration ID gptoss-llamab64739e-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-27T20:07:14Z Notes Equal tensor split; CUDA1 main GPU; complete GPU placement. Applied optimizations GPU layer offload The second card removed the one-card host-weight bottleneck. GPU link topology Layer split limited traffic across the PCIe SYS path. GPU peer access NVIDIA peer access was unavailable on this topology. Same-host layer split Complete model placement across two V100s. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `gptoss-2v100-layer-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T20:07:14Z Statistic median; 5 samples Model gpt-oss-120B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 627.4 tok/s Decode 99.9 tok/s Baseline gos (+951.0%) Evidence results/20260727-new-model-dual-v100/gpt-oss-120b-dual-layer-1-1-mg1.txt Benchmark notes. Complete GPU placement; correctness check pending. Result notes. Decode 10.51 times the control; four samples were near 100 tok/s.
	llama.cpp	CPU + V100	1024	512	—	✕	—	—	27.2	9.5	NA	NA
`gos` gpt-oss-120B fitted to one V100.1 result · 0 techniques · 1 parameter Configuration ID gptoss-llamab64739e-v100-fit1g-b128-c1k-f16 Created 2026-07-27T20:09:00Z Notes CUDA0 with automatic CPU fallback. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 with automatic fit; remaining tensors on CPU. Results and evidence `gptoss-v100-fit-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T20:09:00Z Statistic median; 5 samples Model gpt-oss-120B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 27.2 tok/s Decode 9.5 tok/s Evidence results/20260727-new-model-dual-v100/gpt-oss-120b-single-gpu0.txt Benchmark notes. One-card control; correctness check pending. Result notes. Five-sample medians.

Laguna S 2.1

Size: 118B / 8B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp/poolside	CPU + V100	4096	11	—	✕	—	—	18.5	12.9	NA	NA
`ls-d0` Laguna S target with VRAM reserved for corrected DFlash.1 result · 4 techniques · 6 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe33-b128-c4k-p1-f16-copy-fa-target Created 2026-07-25T07:24:00Z Notes Matched control for the corrected one-token DFlash profile. Applied optimizations CPU affinity The CPU mask was ffff on the dual-socket R720. Flash attention The matched Laguna S target control used flash attention. MoE expert placement Thirty-three early routed-expert blocks stayed on the CPUs to reserve V100 memory for DFlash. Private weight copy The matched target control used --no-mmap. Uncommon parameters backend.server_slots 1 slots system.model_memory_mode private-copy — llama.cpp --no-mmap used the large host RAM. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.cpu_moe_blocks 33 blocks target.gpu_expert_blocks 16 blocks target.gpu_non_expert_blocks 49 blocks Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; Thirty-three early routed-expert blocks on CPU; DFlash VRAM reserved. Results and evidence `laguna-s-dflash-corrected-baseline-20260725` Task text generation Status VALID Measured 2026-07-25T07:24:00Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 18.5 tok/s Decode 12.9 tok/s Evidence laguna/optimization-results/20260725T0724Z-dflash-corrected Benchmark notes. Matched target control with VRAM reserved for corrected DFlash. Result notes. Median of three matched target-only server requests.
	llama.cpp/poolside	CPU + V100	4096	11	—	✕	—	—	31.0	18.5	NA	NA
	llama.cpp/poolside	CPU + V100	4096	4096	—	✕	—	—	123.7	17.6	NA	NA
`ls-s2` Laguna S 4K server with NUMA interleave and Flash Attention.2 results · 6 techniques · 0 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe31-b1024-c4k-p1-f16-copy-fa-interleave Created 2026-07-26T02:15:00Z Notes Retained service profile: numactl --interleave=all, 18 V100 expert blocks, batch and microbatch 1024. Applied optimizations Batch-size tuning Batch and microbatch 1024 raised pp4096 prefill from 72 to 124 tok/s. Microbatch 1024 nearly doubled 4K prefill against microbatch 512. CPU affinity The CPU mask was ffff with strict affinity on the dual-socket R720. Flash attention Flash Attention with F16 KV added about 5% decode and shrank the CUDA compute buffer enough for microbatch 1024. MoE expert placement Thirty-one early routed-expert blocks stayed on the CPUs; 18 expert blocks used the V100. NUMA memory policy Same interleaved placement as the 4K server profile. numactl --interleave=all spread the private no-mmap weights across both R720 memory controllers for a 30% decode gain. Private weight copy The profile kept --no-mmap; interleave requires the private allocation. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; numactl --interleave=all around the process plus --numa distribute. Thirty-one early routed-expert blocks on CPU; all other tensors on V100. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; Same placement as the interleaved 4K server profile. Results and evidence `laguna-s-interleave-fa-c4k-20260726` Task text generation Status VALID Measured 2026-07-26T02:15:00Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 31.0 tok/s Decode 18.5 tok/s Baseline ls-s1 (+37.7%) Output matches baseline yes Evidence laguna/optimization-results/20260726T0200Z-interleave-fa Benchmark notes. Single-slot 4K server under numactl --interleave=all with Flash Attention. Every token of all five samples matched the historical control. Result notes. Medians of five server requests. 37.7% over the prior 4K profile. Every token of every sample matched the historical control. `laguna-s-interleave-fa-pp4096-20260726` Task text generation Status VALID Measured 2026-07-26T01:48:17Z Statistic mean; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 4096 Generated tokens 0 Prefill 123.7 tok/s Decode 17.6 tok/s Evidence laguna/benchmark-laguna-s-poolside-interleave-fa-pp4096-20260725.jsonl Benchmark notes. Full 4K prefill with microbatch 1024 under numactl --interleave=all. The same file holds a 17.568 tok/s tg32 test. Result notes. Mean of three pp4096 runs and the matched shallow tg32 mean from the same llama-bench invocation.
	llama.cpp/poolside	CPU + V100	4096	128	—	✕	—	—	6.5	12.0	NA	NA
`ls-b0` Laguna S Poolside control with automatic fitted tensor placement.1 result · 1 technique · 3 parameters Configuration ID laguna-s-poolside04b2b72-v100-fit4g-b2048-c4k-f16-mmap Created 2026-07-25T06:55:00Z Notes Matched fixed-token control for the Laguna optimization. Applied optimizations Host page cache The mapped target artifact was warm in the R720 host page cache. Uncommon parameters system.model_memory_mode mmap-warm-page-cache — The complete target artifact was in the host page cache. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.tensor_placement automatic-fit-4096-mib-margin MiB Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; NUMA distribute; Automatic tensor-level fit with a 4 GiB V100 reserve. Results and evidence `laguna-s-poolside-bench-baseline-20260725` Task text generation Status VALID Measured 2026-07-25T06:55:00Z Statistic mean; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 128 Generated tokens 32 Prefill 6.5 tok/s Decode 12.0 tok/s Evidence laguna/benchmark-laguna-s-poolside-baseline-20260725.jsonl Benchmark notes. Matched Poolside llama-bench control with a 4 GiB fit reserve. Result notes. Means of three pp128 and tg32 samples.
	llama.cpp/poolside	CPU + V100	TODO	0	—	✕	—	—	—	13.2	NA	NA
`ls-b1` Laguna S shallow decode with 19 V100 expert blocks.1 result · 3 techniques · 5 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe30-b128-minctx-f16-copy Created 2026-07-25T07:17:55Z Notes Peak fixed-token profile. This profile does not support a 4K server. Applied optimizations CPU affinity The CPU mask was ffff on the dual-socket R720. MoE expert placement Thirty early routed-expert blocks stayed on the CPUs; 19 expert blocks used the V100. Private weight copy The Poolside backend used --no-mmap for the CPU-resident expert weights. Uncommon parameters system.model_memory_mode private-copy — llama.cpp --no-mmap used the large host RAM. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.cpu_moe_blocks 30 blocks target.gpu_expert_blocks 19 blocks target.gpu_non_expert_blocks 49 blocks Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; Thirty early routed-expert blocks on CPU; all other tensors on V100. Results and evidence `laguna-s-poolside-ncmoe30-no-mmap-20260725` Task text generation Status VALID Measured 2026-07-25T07:17:55Z Statistic mean; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Prompt tokens 0 Generated tokens 32 Decode 13.2 tok/s Evidence laguna/benchmark-laguna-s-poolside-ncmoe30-no-mmap-20260725.jsonl Benchmark notes. Peak shallow decode with 19 expert blocks on the V100. Result notes. Mean of three shallow tg32 samples.
	llama.cpp/poolside	CPU + V100	4096	11	—	✕	—	—	9.8	13.4	NA	NA
`ls-s1` Laguna S 4K server with 18 V100 expert blocks.1 result · 8 techniques · 13 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe31-b128-c4k-p1-f16-copy Created 2026-07-25T07:19:00Z Notes Retained single-slot service profile with private host weights. Applied optimizations Batch-size tuning Batch and microbatch were both 128 for the retained 4K single-slot service profile. CPU affinity The CPU mask was ffff on the dual-socket R720. GPU layer offload The V100 held all 49 non-expert blocks and 18 expert blocks. MoE expert placement Thirty-one early routed-expert blocks stayed on the CPUs; 18 expert blocks used the V100. Private weight copy The Poolside backend used --no-mmap for the CPU-resident expert weights. Release GPU capacity ComfyUI, the model router, and GPU embeddings were stopped. Runtime selection The retained service profile used the pinned Poolside Laguna branch. Weight quantization The target artifact used Q4_K_M weights. Uncommon parameters accelerator:r720-v100-32gb.clock_policy automatic accelerator:r720-v100-32gb.ecc_mode enabled boolean accelerator:r720-v100-32gb.free_vram_at_decode 1051 MiB MiB — Observed after the correctness samples. accelerator:r720-v100-32gb.persistence_mode disabled boolean backend.cuda_compute_graph 663.5 MiB MiB — Observed allocation that prevented the 19-block profile. backend.server_slots 1 slots node:r720.resident_set_size 43130880 KiB KiB — Observed after model load and the correctness samples. system.cpu_frequency_governor schedutil system.model_memory_mode private-copy — llama.cpp --no-mmap used the large host RAM. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.cpu_moe_blocks 31 blocks target.gpu_expert_blocks 18 blocks target.gpu_non_expert_blocks 49 blocks Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; Thirty-one early routed-expert blocks on CPU; all other tensors on V100. Results and evidence `laguna-s-poolside-ncmoe31-no-mmap-c4k-20260725` Task text generation Status VALID Measured 2026-07-25T07:19:00Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 9.8 tok/s Decode 13.4 tok/s Output matches baseline yes Evidence laguna/optimization-results/20260725T0719Z-target-ncmoe31-no-mmap Benchmark notes. Single-slot 4K server with 18 expert blocks on the V100. Result notes. Medians of three server requests. Every token matched the historical control.
	llama.cpp/laguna	CPU + V100	4096	128	—	✕	—	—	9.6	7.7	NA	NA
`ls-g` Legacy configuration for laguna-s-v1001 result · 0 techniques · 0 parameters Configuration ID laguna-s-v100 Created 2026-07-23T10:19:52Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute Results and evidence `laguna-s-v100` Task text generation Status VALID Measured 2026-07-23T10:19:52Z Statistic mean; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 128 Generated tokens 32 Prefill 9.6 tok/s Decode 7.7 tok/s Evidence laguna/benchmark-laguna-s-v100.jsonl Benchmark notes. Bonsai and GPU vsearch unloaded. --fit-target 1024 assigned the 70.01 GiB model across CPU and V100 memory. Result notes. Mean of three pp128 and tg32 samples with fitted CPU and V100 placement.
	llama.cpp/poolside	CPU + V100	4096	11	—	✕	—	—	14.5	12.5	NA	NA
`ls-pb` Legacy configuration for laguna-s-v100-dflash-baseline1 result · 0 techniques · 0 parameters Configuration ID laguna-s-v100-dflash-baseline Created 2026-07-24T17:19:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute Results and evidence `laguna-s-v100-dflash-baseline` Task text generation Status VALID Measured 2026-07-24T17:19:00Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 14.5 tok/s Decode 12.5 tok/s Evidence laguna/dflash-results/20260724T172118Z/summary.txt Benchmark notes. Matched baseline with a 4 GiB V100 reserve. Result notes. Median of three 11-token prompt and 32-token decode samples.
	llama.cpp/poolside	CPU + V100(2)	131072	128221	—	✕	—	—	170.6	12.6	NA	NA
`ls2c128` Laguna S repository coding task with a 128.2K-token prompt.1 result · 0 techniques · 3 parameters Configuration ID laguna-s-poolside04b2b72-2v100-autofit-c128k-code-p128221 Created 2026-07-27T04:06:44Z Notes 128K server; automatic tensor fit; reasoning and prompt cache off. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.prompt_cache off backend.reasoning off target.max_output_tokens 2500 tokens Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 split; automatic 128K tensor fit with CPU fallback. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 split and main GPU; automatic 128K tensor fit. Results and evidence `laguna-s-2v100-code128k-20260727` Task text generation Status VALID Measured 2026-07-27T04:06:44Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 0 Prompt tokens 128221 Generated tokens 1565 Prefill 170.6 tok/s Decode 12.6 tok/s Elapsed 875.886 s Evidence results/20260727-long-context/coding-128k-scored.response.json Benchmark notes. Repository scheduler implementation; passed all eleven tests. Result notes. Complete replacement file passed 11/11 retained tests.
	llama.cpp/poolside	CPU + V100(2)	131072	29546	—	✕	—	—	180.7	19.3	NA	NA
`ls2c32` Laguna S repository coding task with a 29.5K-token prompt.1 result · 0 techniques · 3 parameters Configuration ID laguna-s-poolside04b2b72-2v100-autofit-c128k-code-p29546 Created 2026-07-27T03:56:12Z Notes 128K server; automatic tensor fit; reasoning and prompt cache off. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.prompt_cache off backend.reasoning off target.max_output_tokens 2500 tokens Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 split; automatic 128K tensor fit with CPU fallback. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 split and main GPU; automatic 128K tensor fit. Results and evidence `laguna-s-2v100-code32k-20260727` Task text generation Status VALID Measured 2026-07-27T03:56:12Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 0 Prompt tokens 29546 Generated tokens 1168 Prefill 180.7 tok/s Decode 19.3 tok/s Elapsed 224.073 s Evidence results/20260727-long-context/coding-32k-scored.response.json Benchmark notes. Repository scheduler implementation; passed all eleven tests. Result notes. Complete replacement file passed 11/11 retained tests.
	llama.cpp/poolside	CPU + V100(2)	131072	62067	—	✕	—	—	179.2	18.6	NA	NA
`ls2c64` Laguna S repository coding task with a 62.1K-token prompt.1 result · 0 techniques · 3 parameters Configuration ID laguna-s-poolside04b2b72-2v100-autofit-c128k-code-p62067 Created 2026-07-27T04:00:16Z Notes 128K server; automatic tensor fit; reasoning and prompt cache off. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.prompt_cache off backend.reasoning off target.max_output_tokens 2500 tokens Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 split; automatic 128K tensor fit with CPU fallback. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 split and main GPU; automatic 128K tensor fit. Results and evidence `laguna-s-2v100-code64k-20260727` Task text generation Status VALID Measured 2026-07-27T04:00:16Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 0 Prompt tokens 62067 Generated tokens 155 Prefill 179.2 tok/s Decode 18.6 tok/s Elapsed 354.748 s Evidence results/20260727-long-context/coding-64k-scored.response.json Benchmark notes. Repository scheduler implementation; copied the incomplete stub. Result notes. Response copied the incomplete stub and passed 0/11 tests.
	llama.cpp/poolside	CPU + V100(2)	8192	128	—	✕	—	—	62.0	30.6	NA	NA
	llama.cpp/poolside	CPU + V100(2)	8192	128	4096	✕	—	—	65.9	28.0	NA	NA
`ls2ctx` Laguna S dual-V100 context-depth sweep.2 results · 0 techniques · 2 parameters Configuration ID laguna-s-poolside04b2b72-2v100-ncmoe12-b1024-c8k-f16-depth Created 2026-07-27T02:48:09Z Notes Twelve CPU expert blocks; equal layer split; 8K context allocation. Applied optimizations No applied technique is cataloged. Uncommon parameters target.cpu_expert_blocks 12 blocks target.fit_min_ctx 8192 tokens Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 split; twelve CPU expert blocks. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 split and main GPU. Results and evidence `laguna-s-2v100-depth4096-20260727` Task text generation Status VALID Measured 2026-07-27T02:50:07Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 8192 Tested depth 4096 Prompt tokens 128 Generated tokens 32 Prefill 65.9 tok/s Decode 28.0 tok/s Evidence results/20260726-dual-v100/laguna-dual-depth-0-4k-ncmoe12.txt Benchmark notes. One decode sample fell to 20.877 tok/s; four were 26.480-28.335. Result notes. Decode retains 91.7% of the zero-depth median. `laguna-s-2v100-depth0-20260727` Task text generation Status VALID Measured 2026-07-27T02:48:09Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 8192 Tested depth 0 Prompt tokens 128 Generated tokens 32 Prefill 62.0 tok/s Decode 30.6 tok/s Evidence results/20260726-dual-v100/laguna-dual-depth-0-4k-ncmoe12.txt Benchmark notes. Zero-depth control with twelve CPU expert blocks. Result notes. Decode samples: 24.095, 29.394, 30.563, 30.626, 30.561 tok/s.
	llama.cpp/poolside	CPU + V100(2)	TODO	0	—	✕	—	—	—	32.5	NA	NA
	llama.cpp/poolside	CPU + V100(2)	TODO	0	—	✕	—	—	—	32.2	NA	NA
	llama.cpp/poolside	CPU + V100(2)	4096	11	—	✕	—	—	30.2	25.4	NA	NA
`l2fit` Laguna S fitted across two local V100s and host experts.3 results · 1 technique · 6 parameters Configuration ID laguna-s-poolside04b2b72-2v100-fit1g-layer-b1024-f16-fa-copy Created 2026-08-29T20:50:21Z Notes The fitter retained 10.89 GiB of expert weights in CUDA_Host memory. Applied optimizations Same-host layer split The two local V100s used layer split as the matched control. Uncommon parameters system.prefix_cache_state not-applicable — llama-bench had zero prompt tokens; raw correctness mode disabled caching. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for the run. target.cuda_host_expert_weights 10891.69 MiB — Blocks 41 through 47 and part of block 40. target.fit enabled target.fit_target 1024 MiB per GPU target.tensor_split 1,1 fitted Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 25 GPU layers; 16 threads; 16 batch threads; affinity ffff; CUDA0 and traced host expert execution. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 25 GPU layers; 16 threads; 16 batch threads; affinity ffff; CUDA0 and host expert execution. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 25 GPU layers; 16 threads; 16 batch threads; affinity ffff; Matched correctness control. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 24 GPU layers; 16 threads; 16 batch threads; affinity ffff; CUDA1 and main GPU. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 24 GPU layers; 16 threads; 16 batch threads; affinity ffff; Matched correctness control; main GPU. Results and evidence `laguna-s-2v100-matched-completion-20260829` Task text generation Status PROVISIONAL Measured 2026-08-29T21:51:31Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 64 Prefill 30.2 tok/s Decode 25.4 tok/s Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/local-ai-laguna-matched-2v100.log Benchmark notes. Single raw-completion correctness control; not a retained throughput statistic. Start time was reconstructed from the output mtime and the logged process elapsed time. Result notes. Single correctness request. `laguna-s-2v100-fit-audit-20260829` Task text generation Status PROVISIONAL Measured 2026-08-29T21:00:09Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 64 Decode 32.5 tok/s Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/two-v100-control-audit.json Benchmark notes. Trace-enabled diagnostic run. Samples: 32.7206, 29.8702, 32.5007 tok/s. Provisional because tracing produced one low outlier. Result notes. Three-sample traced median. `laguna-s-2v100-fit-control-20260829` Task text generation Status VALID Measured 2026-08-29T20:50:21Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 32.2 tok/s Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/two-v100-control.json Benchmark notes. Zero-prompt decode. Samples: 32.0916, 32.1608, 32.1914, 32.0755, 32.2798 tok/s. Result notes. Five-sample decode median.
	llama.cpp/poolside	CPU + V100(2)	4096	0	—	✕	—	—	—	32.1	NA	NA
`ls2x` Laguna S with 10 CPU expert blocks and two V100s.1 result · 1 technique · 2 parameters Configuration ID laguna-s-poolside04b2b72-2v100-ncmoe10-layer-1-1-mg1-b1024-f16-copy-fa Created 2026-07-27T01:32:11Z Notes Fit-boundary test with 39 GPU expert blocks; slower than ncmoe12. Applied optimizations MoE expert placement Two additional GPU expert blocks reduced decode slightly. Uncommon parameters target.cpu_expert_blocks 10 blocks target.gpu_expert_blocks 39 blocks Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 part of the layer split; 10 CPU expert blocks. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 part of the layer split and main GPU. Results and evidence `laguna-s-2v100-ncmoe10-20260727` Task text generation Status VALID Measured 2026-07-27T01:32:11Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 0 Generated tokens 32 Decode 32.1 tok/s Baseline ls1r (+77.4%) Output matches baseline yes Evidence results/20260726-dual-v100/laguna-dual-layer-1-1-mg1-ncmoe10.txt Benchmark notes. Fit-boundary test with 39 GPU expert blocks. Result notes. Decode +77.4%; slower than the 12-CPU-expert placement.
	llama.cpp/poolside	CPU + V100(2)	4096	0	—	✕	—	—	—	32.2	NA	NA
`ls2r` Laguna S with 12 CPU expert blocks and two V100s.1 result · 4 techniques · 2 parameters Configuration ID laguna-s-poolside04b2b72-2v100-ncmoe12-layer-1-1-mg1-b1024-f16-copy-fa Created 2026-07-27T01:31:15Z Notes Retained dual-V100 placement with 37 GPU expert blocks. Applied optimizations GPU link topology Layer split limited transfers across the PCIe SYS path. GPU peer access NVIDIA reported peer reads and writes as topology not supported. MoE expert placement Twelve CPU expert blocks beat the fit-boundary ten-block placement. NUMA memory policy Private CPU weights were interleaved across both memory controllers. Uncommon parameters target.cpu_expert_blocks 12 blocks target.gpu_expert_blocks 37 blocks Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 part of the layer split; 12 CPU expert blocks. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 part of the layer split and main GPU. Results and evidence `laguna-s-2v100-ncmoe12-20260727` Task text generation Status VALID Measured 2026-07-27T01:31:15Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 0 Generated tokens 32 Decode 32.2 tok/s Baseline ls1r (+78.2%) Output matches baseline yes Evidence results/20260726-dual-v100/laguna-dual-layer-1-1-mg1-ncmoe12.txt Benchmark notes. Private interleaved copy with 37 GPU expert blocks. Result notes. Decode +78.2%; byte-identical fixed-seed output.
	llama.cpp/poolside	CPU + V100(2)	4096	0	—	✕	—	—	—	18.1	NA	NA
`ls1r` Laguna S with 31 CPU expert blocks and one V100.1 result · 0 techniques · 0 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe31-b1024-f16-copy-fa Created 2026-07-27T01:29:55Z Notes Matched rerun of the retained one-V100 placement. Applied optimizations No applied technique is cataloged. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; Thirty-one CPU expert blocks; CUDA0 only. Results and evidence `laguna-s-v100-ncmoe31-rerun-20260727` Task text generation Status VALID Measured 2026-07-27T01:29:55Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 0 Generated tokens 32 Decode 18.1 tok/s Evidence results/20260726-dual-v100/laguna-single-gpu0-ncmoe31.txt Benchmark notes. Private interleaved weight copy with 18 GPU expert blocks. Result notes. Five-sample median; matched one-card placement.
	llama.cpp/poolside	CPU + V100(2)	131072	128	32768	✕	—	—	62.2	23.6	NA	NA
	llama.cpp/poolside	CPU + V100(2)	131072	128	65536	✕	—	—	55.5	17.6	NA	NA
	llama.cpp/poolside	CPU + V100(2)	131072	128	131072	✕	—	—	46.8	15.4	NA	NA
`ls2long` Laguna S dual-V100 native long-context depth sweep.3 results · 0 techniques · 2 parameters Configuration ID laguna-s-poolside04b2b72-2v100-ncmoe12-b1024-c128k-f16-depth Created 2026-07-27T03:17:46Z Notes Twelve CPU expert blocks; 128K fitted context; F16 KV. Applied optimizations No applied technique is cataloged. Uncommon parameters target.cpu_expert_blocks 12 blocks target.fit_min_ctx 131072 tokens Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 split; twelve CPU expert blocks. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 split and main GPU; twelve CPU expert blocks. Results and evidence `laguna-s-2v100-depth131072-long-20260727` Task text generation Status VALID Measured 2026-07-27T03:29:38Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 131072 Prompt tokens 128 Generated tokens 32 Prefill 46.8 tok/s Decode 15.4 tok/s Evidence results/20260727-long-context/laguna-depth-32k-64k-128k-f16-ncmoe12.txt Benchmark notes. Native-context depth run with twelve CPU expert blocks. Result notes. Three-sample medians; decode samples 14.839, 15.469, 15.394. `laguna-s-2v100-depth65536-long-20260727` Task text generation Status VALID Measured 2026-07-27T03:22:13Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 65536 Prompt tokens 128 Generated tokens 32 Prefill 55.5 tok/s Decode 17.6 tok/s Evidence results/20260727-long-context/laguna-depth-32k-64k-128k-f16-ncmoe12.txt Benchmark notes. Native-context depth run with twelve CPU expert blocks. Result notes. Three-sample medians; decode samples 19.524, 15.625, 17.578. `laguna-s-2v100-depth32768-long-20260727` Task text generation Status VALID Measured 2026-07-27T03:17:46Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 32768 Prompt tokens 128 Generated tokens 32 Prefill 62.2 tok/s Decode 23.6 tok/s Evidence results/20260727-long-context/laguna-depth-32k-64k-128k-f16-ncmoe12.txt Benchmark notes. Native-context depth run with twelve CPU expert blocks. Result notes. Three-sample medians; decode samples 23.326, 23.643, 23.650.
	llama.cpp/poolside	V100(3)	4096	171	—	✕	—	—	120.2	37.7	NA	NA
`l3srv` Laguna S corrected three-V100 server with a cache miss.1 result · 3 techniques · 7 parameters Configuration ID laguna-s-poolside04b2b72-3v100-rpc-server331-cache-miss-c4k Created 2026-08-29T21:40:05Z Notes Server device order CUDA0,RPC0,CUDA1 used shares 3:1:3. Applied optimizations Cross-host layer split Two local V100 shares and one c720 V100 share used RPC TCP. GPU link topology RPC used the direct 40 Gb/s ConnectX-3 rail measured at 39.395 Gb/s. MoE expert placement The third V100 removed CPU expert execution; the uneven split was the retained placement. Uncommon parameters backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP iperf, 40 Gb/s link, MTU 9000, zero retransmits. system.prefix_cache_state miss cached tokens system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for the run. target.fit disabled target.model_buffers 30451.56,30737.18,10332.97 MiB — CUDA0, CUDA1, RPC0. target.tensor_split 3,1,3 in CUDA0,RPC0,CUDA1 order — Equivalent uneven placement for llama-server device order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Corrected server local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Corrected server local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 7 GPU layers; 6 threads; network enp65s0d1; Corrected server RPC0 share. Results and evidence `laguna-s-3v100-coding-trial-20260829` Task text generation Status PROVISIONAL Measured 2026-08-29T21:42:14Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 171 Generated tokens 418 Prefill 120.2 tok/s Decode 37.7 tok/s Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/trial-coding-response.json Benchmark notes. Short-prompt qualitative screen, not an agent workload. The interval-merging repair was manually checked and correct. Result notes. Manually checked correct coding response.
	llama.cpp/poolside	V100(3)	4096	11	—	✕	—	—	26.6	37.3	NA	NA
	llama.cpp/poolside	V100(3)	TODO	0	—	✕	—	—	—	38.9	NA	NA
	llama.cpp/poolside	V100(3)	TODO	0	—	✕	—	—	—	38.9	NA	NA
`l3m31` Laguna S manual 21:21:7 cross-host layer split.3 results · 3 techniques · 7 parameters Configuration ID laguna-s-poolside04b2b72-3v100-rpc-manual331-b1024-f16-fa-copy Created 2026-08-29T20:59:05Z Notes Fit disabled; tensor shares 3:3:1 over CUDA0, CUDA1, and RPC0. Applied optimizations Cross-host layer split Two local V100 shares and one c720 V100 share used RPC TCP. GPU link topology RPC used the direct 40 Gb/s ConnectX-3 rail measured at 39.395 Gb/s. MoE expert placement The third V100 removed CPU expert execution; the uneven split was the retained placement. Uncommon parameters backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP iperf, 40 Gb/s link, MTU 9000, zero retransmits. system.prefix_cache_state not-applicable — Zero-prompt llama-bench decode; raw correctness mode disabled caching. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for the run. target.fit disabled target.model_buffers 30451.56,31125.42,9944.74 MiB — CUDA0, CUDA1, RPC0. target.tensor_split 3,3,1; 21,21,7 layers Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Matched completion local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Manual local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Matched completion local CUDA1 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Manual local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 7 GPU layers; 6 threads; network enp65s0d1; Matched completion RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 7 GPU layers; 6 threads; network enp65s0d1; Manual RPC0 share. Results and evidence `laguna-s-3v100-matched-completion-20260829` Task text generation Status PROVISIONAL Measured 2026-08-29T21:52:42Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 11 Generated tokens 64 Prefill 26.6 tok/s Decode 37.3 tok/s Baseline l2fit (+46.8%) Output matches baseline yes Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/local-ai-laguna-matched-3v100.log Benchmark notes. Single raw-completion correctness arm. Its output matched the two-card control byte for byte. Start time was reconstructed from the output mtime and the logged process elapsed time. Result notes. The 64-token output matched byte for byte. `laguna-s-3v100-rpc-manual-331-20260829` Task text generation Status VALID Measured 2026-08-29T21:04:24Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 38.9 tok/s Baseline l2fit (+21.0%) Output matches baseline yes Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/three-v100-direct-rpc-retained.json Benchmark notes. Retained winner. Samples: 38.9935, 38.6750, 39.0159, 38.9215, 38.7984 tok/s. Result notes. Five-sample median; 21.0% faster than the control. `laguna-s-3v100-rpc-manual-331-screen-20260829` Task text generation Status VALID Measured 2026-08-29T20:59:05Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 64 Decode 38.9 tok/s Baseline l2fit (+20.9%) Output matches baseline yes Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/three-v100-split-3-3-1-manual-screen.json Benchmark notes. Passed screen with 21:21:7 layers. Median 38.8887 tok/s. Result notes. Three-sample screen; matched completion checked separately.
	llama.cpp/poolside	V100(3)	4096	133	—	✕	—	—	143.3	39.1	NA	NA
`l3srp` Laguna S corrected server with a partial prefix-cache hit.1 result · 3 techniques · 7 parameters Configuration ID laguna-s-poolside04b2b72-3v100-rpc-server331-cache-partial-c4k Created 2026-08-29T21:42:52Z Notes Same placement as l3srv; 33 of 133 prompt tokens were cached. Applied optimizations Cross-host layer split Two local V100 shares and one c720 V100 share used RPC TCP. GPU link topology RPC used the direct 40 Gb/s ConnectX-3 rail measured at 39.395 Gb/s. MoE expert placement The third V100 removed CPU expert execution; the uneven split was the retained placement. Uncommon parameters backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP iperf, 40 Gb/s link, MTU 9000, zero retransmits. system.prefix_cache_state partial-hit cached tokens — Thirty-three of 133 prompt tokens were cached. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for the run. target.fit disabled target.model_buffers 30451.56,30737.18,10332.97 MiB — CUDA0, CUDA1, RPC0. target.tensor_split 3,1,3 in CUDA0,RPC0,CUDA1 order — Equivalent uneven placement for llama-server device order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Corrected server local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 21 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Corrected server local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 7 GPU layers; 6 threads; network enp65s0d1; Corrected server RPC0 share. Results and evidence `laguna-s-3v100-reasoning-trial-20260829` Task text generation Status INVALID Measured 2026-08-29T21:42:52Z Statistic single; 1 sample Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 133 Generated tokens 307 Prefill 143.3 tok/s Decode 39.1 tok/s Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/trial-reasoning-response.json Benchmark notes. Failed short-prompt unit-conversion screen. Thirty-three prompt tokens were served from the prefix cache. Result notes. Timing is retained, but the task failed.
	llama.cpp/poolside	V100(3)	TODO	0	—	✕	—	—	—	26.4	NA	NA
`l3eq` Laguna S automatically fitted across three V100s.1 result · 2 techniques · 7 parameters Configuration ID laguna-s-poolside04b2b72-3v100-rpc-fit1g-equal-b1024-f16-fa-copy Created 2026-08-29T20:53:17Z Notes Two V100s were local and RPC0 was the c720 V100. Applied optimizations Cross-host layer split Two local V100 shares and one c720 V100 share used RPC TCP. GPU link topology RPC used the direct 40 Gb/s ConnectX-3 rail measured at 39.395 Gb/s. Uncommon parameters backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP iperf, 40 Gb/s link, MTU 9000, zero retransmits. system.prefix_cache_state not-applicable — Zero-prompt llama-bench decode. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for the run. target.fit enabled target.fit_target 1024 MiB per GPU target.tensor_split automatic equal Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 17 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 16 GPU layers; 6 threads; network enp65s0d1; RPC0 share on the remote V100. Results and evidence `laguna-s-3v100-rpc-equal-fit-20260829` Task text generation Status VALID Measured 2026-08-29T20:53:17Z Statistic median; 5 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 26.4 tok/s Baseline l2fit (-18.0%) Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/three-v100-direct-rpc.json Benchmark notes. Rejected automatic placement. Median 26.3659 tok/s; mean 26.408758 tok/s. Result notes. Automatic equal placement lost 18.0%.
	llama.cpp/poolside	V100(3)	TODO	0	—	✕	—	—	—	25.6	NA	NA
`l3f31` Laguna S with 3:3:1 requested while fitting stayed enabled.1 result · 2 techniques · 8 parameters Configuration ID laguna-s-poolside04b2b72-3v100-rpc-fit1g-request331-b1024-f16-fa-copy Created 2026-08-29T20:56:52Z Notes The fitter replaced the request with approximately equal placement. Applied optimizations Cross-host layer split Two local V100 shares and one c720 V100 share used RPC TCP. GPU link topology RPC used the direct 40 Gb/s ConnectX-3 rail measured at 39.395 Gb/s. Uncommon parameters backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP iperf, 40 Gb/s link, MTU 9000, zero retransmits. system.prefix_cache_state not-applicable — Zero-prompt llama-bench decode. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for the run. target.fit enabled target.fit_target 1024 MiB per GPU target.model_buffers 24392.12,23840.23,23289.37 MiB — CUDA0, CUDA1, RPC0. target.tensor_split 3,3,1 requested; fit produced 17,16,16 layers — The requested shares did not survive fitting. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 17 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Fit replaced the requested local share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; network roce0; Fit replaced the requested local share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 16 GPU layers; 6 threads; network enp65s0d1; Fit expanded the requested remote share to 16 layers. Results and evidence `laguna-s-3v100-rpc-fit-request-331-20260829` Task text generation Status VALID Measured 2026-08-29T20:56:52Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 64 Decode 25.6 tok/s Baseline l2fit (-20.4%) Evidence results/20260829T204813Z-laguna-s-direct-rpc-3v100/three-v100-split-3-3-1-screen.json Benchmark notes. Rejected screen. Fit replaced requested 3:3:1 with 17:16:16 layers. Median 25.5891 tok/s. Result notes. Fitting erased the requested uneven placement.
	llama.cpp/poolside	CPU + V100	4096	11	—	Local GPU DFlash	+2.0%	66.7%	20.2	13.2	NA	NA
`ls-d1` Laguna S with corrected one-token DFlash on the V100.1 result · 9 techniques · 11 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe33-b128-c4k-p1-f16-copy-fa-dflash-w1 Created 2026-07-25T07:29:00Z Notes Corrected draft SHA-256 starts with 2ee8aa30338d. Applied optimizations Batch-size tuning Batch and microbatch were both 128 to leave V100 capacity for the draft model. CPU affinity The CPU mask was ffff on the dual-socket R720. Draft placement The corrected DFlash draft and selected target tensors used the V100. Flash attention The corrected DFlash benchmark used flash attention. MoE expert placement Thirty-three early routed-expert blocks stayed on the CPUs; 16 expert blocks and DFlash used the V100. Private weight copy The target and corrected DFlash benchmark used --no-mmap. Release GPU capacity ComfyUI, the model router, and GPU embeddings were stopped. Speculative decoding The matching DFlash model proposed tokens for target verification. Speculative policy tuning The corrected DFlash result used one proposed token. Uncommon parameters accelerator:r720-v100-32gb.clock_policy automatic accelerator:r720-v100-32gb.ecc_mode enabled boolean accelerator:r720-v100-32gb.free_vram_at_decode 1633 MiB MiB — Observed with corrected DFlash loaded. backend.server_slots 1 slots draft.weight_placement full-v100 draft.weight_revision corrected-2ee8aa30338d — Poolside corrected DFlash GGUF. system.model_memory_mode private-copy — llama.cpp --no-mmap used the large host RAM. system.resident_gpu_services unloaded — ComfyUI, model router, and GPU embeddings released VRAM. target.cpu_moe_blocks 33 blocks target.gpu_expert_blocks 16 blocks target.gpu_non_expert_blocks 49 blocks Node placement r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA distribute; Same target placement; corrected DFlash is fully on the V100. Results and evidence `laguna-s-dflash-corrected-w1-20260725` Task text generation Status VALID Measured 2026-07-25T07:29:00Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 20.2 tok/s Decode 13.2 tok/s Baseline ls-d0 (+2.0%) Output matches baseline yes Evidence laguna/optimization-results/20260725T0724Z-dflash-corrected Benchmark notes. Corrected DFlash on V100 with a one-token policy. Result notes. Median of three corrected DFlash requests; all token sequences matched.
	llama.cpp/poolside	CPU + V100	4096	11	—	Local GPU DFlash	-6.4%	66.7%	28.8	17.3	NA	NA
`ls-d2` Laguna S one-token DFlash on the interleaved base.1 result · 3 techniques · 0 parameters Configuration ID laguna-s-poolside04b2b72-v100-ncmoe33-b512-c4k-p1-f16-copy-fa-interleave-dflash-w1 Created 2026-07-26T01:40:00Z Notes Slower than ls-s2; retained as the DFlash comparison point. Applied optimizations NUMA memory policy Same interleaved placement with 33 CPU expert blocks for draft VRAM. Speculative decoding The matching DFlash model proposed tokens for target verification. Speculative policy tuning One proposed token; the profile still lost 6.4% to target-only. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 49 target layers; 49 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; Thirty-three early routed-expert blocks on CPU; corrected DFlash fully on the V100. Results and evidence `laguna-s-interleave-fa-dflash-w1-20260726` Task text generation Status VALID Measured 2026-07-26T01:40:00Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 28.8 tok/s Decode 17.3 tok/s Baseline ls-s2 (-6.4%) Output matches baseline yes Evidence laguna/optimization-results/20260726T0200Z-interleave-fa/dflash-w1 Benchmark notes. One-token DFlash on the interleaved Flash Attention base. Slower than the target-only profile, so speculation stays off. Result notes. Median of three DFlash requests against the 18.497527 tok/s interleaved target-only profile. All token sequences matched.
	llama.cpp/poolside	CPU + V100	4096	11	—	Local GPU DFlash	-41.2%	18.4%	16.6	7.4	NA	NA
`ls-d7` Legacy configuration for laguna-s-v100-dflash-w71 result · 2 techniques · 0 parameters Configuration ID laguna-s-v100-dflash-w7 Created 2026-07-24T17:20:00Z Notes Backfilled configuration key. See benchmark and benchmark_node for typed dimensions. Applied optimizations Speculative decoding The matching DFlash model proposed tokens for target verification. Speculative policy tuning The provisional DFlash result used seven proposed tokens. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA distribute Results and evidence `laguna-s-v100-dflash-w7` Task text generation Status PROVISIONAL Measured 2026-07-24T17:20:00Z Statistic median; 3 samples Model Laguna S 2.1 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Prompt tokens 11 Generated tokens 32 Prefill 16.6 tok/s Decode 7.4 tok/s Baseline ls-pb (-41.2%) Output matches baseline yes Evidence laguna/dflash-results/20260724T172118Z/summary.txt Benchmark notes. DFlash fully offloaded with a 4 GiB V100 target reserve. Provisional because Poolside later corrected the draft GGUF config. Result notes. Median of three samples. All outputs matched the baseline.

Mistral Small 4 119B 2603

Size: 119B
Quant: UD-IQ3_XXS

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(2)	1024	512	—	✕	—	—	422.6	75.1	—	NA
`ms4d` Mistral Small 4 split by layer across two V100s.1 result · 4 techniques · 2 parameters Configuration ID mistral4-llamab64739e-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-27T06:04:56Z Notes Equal tensor split; CUDA1 main GPU; complete GPU placement. Applied optimizations GPU layer offload The second card removed the one-card host-weight bottleneck. GPU link topology Layer split limited traffic across the PCIe SYS path. GPU peer access NVIDIA peer access was unavailable on this topology. Same-host layer split Complete model placement across the two local V100s. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `mistral4-2v100-layer-20260727` Task text generation Status VALID Measured 2026-07-27T06:04:56Z Statistic median; 5 samples Model Mistral Small 4 119B 2603 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 422.6 tok/s Decode 75.1 tok/s Baseline ms4s (+525.4%) Output matches baseline yes Evidence results/20260726-mistral-small4-dual-v100/mistral-small4-dual-layer-1-1-mg1.txt Benchmark notes. Complete GPU placement; fixed-seed output matched the control. Result notes. Decode 6.25 times the control; byte-identical fixed-seed output.
	llama.cpp	V100(2)	1024	512	—	✕	—	—	271.5	35.2	—	NA
`ms4row` Mistral Small 4 row split across two V100s.1 result · 1 technique · 2 parameters Configuration ID mistral4-llamab64739e-2v100-row-1-1-mg1-b128-c1k-f16 Created 2026-07-27T06:38:22Z Notes Matched split-mode comparison on the PCIe SYS path. Applied optimizations Tensor parallelism Row split lost to layer split on the non-peer PCIe SYS path. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `mistral4-2v100-row-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T06:38:22Z Statistic median; 5 samples Model Mistral Small 4 119B 2603 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 271.5 tok/s Decode 35.2 tok/s Baseline ms4d (-53.1%) Evidence results/20260726-mistral-small4-dual-v100/mistral-small4-row-tensor-sweep.txt Benchmark notes. Row split; correctness comparison was not retained. Result notes. Row split was 53.1% slower for decode than layer split.
	llama.cpp	V100(2)	1024	512	—	?	—	—	—	—	—	NA
`ms4ten` Mistral Small 4 experimental tensor split.1 result · 1 technique · 2 parameters Configuration ID mistral4-llamab64739e-2v100-tensor-1-1-mg1-b128-c1k-f16 Created 2026-07-27T06:39:15Z Notes Load failed in llama.cpp 9190 before a timed sample. Applied optimizations Tensor parallelism Experimental tensor split failed during Mistral 4 model loading. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `mistral4-2v100-tensor-invalid-20260727` Task text generation Status INVALID Measured 2026-07-27T06:39:15Z Statistic single; 1 sample Model Mistral Small 4 119B 2603 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Evidence results/20260726-mistral-small4-dual-v100/mistral-small4-row-tensor-sweep.txt Benchmark notes. Start time estimated from the retained row/tensor log order. Result notes. Rejected configuration retained from the row/tensor sweep.
	llama.cpp	CPU + V100	1024	512	—	✕	—	—	44.2	12.0	NA	NA
`ms4s` Mistral Small 4 fitted to one V100.1 result · 0 techniques · 1 parameter Configuration ID mistral4-llamab64739e-v100-fit1g-b128-c1k-f16 Created 2026-07-27T06:06:11Z Notes CUDA0 control with automatic CPU fallback. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 with automatic fit; remaining tensors on CPU. Results and evidence `mistral4-v100-fit-20260727` Task text generation Status VALID Measured 2026-07-27T06:06:11Z Statistic median; 5 samples Model Mistral Small 4 119B 2603 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 44.2 tok/s Decode 12.0 tok/s Evidence results/20260726-mistral-small4-dual-v100/mistral-small4-single-gpu0.txt Benchmark notes. One-card matched control; fixed-seed output retained separately. Result notes. Five-sample medians; matched one-card placement.
	llama.cpp	CPU + V100	TODO	128	32768	✕	—	—	110.1	48.9	NA	NA
	llama.cpp	V100(2)	TODO	128	131072	✕	—	—	24.5	24.9	NA	NA
`ms4long` Mistral Small 4 dual-V100 context-depth sweep.2 results · 0 techniques · 2 parameters Configuration ID mistral4-llamab64739e-2v100-layer-1-1-mg1-b128-f16-depth Created 2026-07-27T06:40:18Z Notes F16 KV; 128-token prompt increment and 32 generated tokens. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `mistral4-2v100-depth131072-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T06:45:07Z Statistic median; 3 samples Model Mistral Small 4 119B 2603 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 131072 Prompt tokens 128 Generated tokens 32 Prefill 24.5 tok/s Decode 24.9 tok/s Evidence results/20260726-mistral-small4-dual-v100/mistral-small4-depth-32k-128k-256k-f16.txt Benchmark notes. Three-sample native-context depth point. Result notes. Three-sample medians; decode samples 24.4781, 24.9164, 24.9137. `mistral4-2v100-depth32768-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T06:40:18Z Statistic median; 3 samples Model Mistral Small 4 119B 2603 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 32768 Prompt tokens 128 Generated tokens 32 Prefill 110.1 tok/s Decode 48.9 tok/s Evidence results/20260726-mistral-small4-dual-v100/mistral-small4-depth-32k-128k-256k-f16.txt Benchmark notes. Three-sample native-context depth point. Result notes. Three-sample medians; decode samples 47.3565, 48.9233, 49.0175.

Qwen3.5 122B-A10B

Size: 125B / 10B
Quant: Q3_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(2)	1024	512	—	✕	—	—	240.1	44.3	4%	NA
`q35d` Qwen3.5 122B-A10B split by layer across two V100s.1 result · 4 techniques · 2 parameters Configuration ID qwen35-llamab64739e-2v100-layer-1-1-mg1-b128-c1k-f16 Created 2026-07-27T20:00:18Z Notes Equal tensor split; CUDA1 main GPU; complete GPU placement. Applied optimizations GPU layer offload The second card removed the one-card host-weight bottleneck. GPU link topology Layer split limited traffic across the PCIe SYS path. GPU peer access NVIDIA peer access was unavailable on this topology. Same-host layer split Complete model placement across two V100s. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. target.tensor_split 1/1 — CUDA1 was the main GPU. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 part of the two-card placement. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA1 part of the two-card placement and main GPU. Results and evidence `qwen35-2v100-layer-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T20:00:18Z Statistic median; 5 samples Model Qwen3.5 122B-A10B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 240.1 tok/s Decode 44.3 tok/s Baseline q35s (+411.5%) Evidence results/20260727-new-model-dual-v100/qwen35-dual-layer-1-1-mg1.txt Benchmark notes. Complete GPU placement; correctness check pending. Result notes. Decode 5.11 times the one-card control.
	llama.cpp	CPU + V100	1024	512	—	✕	—	—	27.3	8.7	NA	NA
`q35s` Qwen3.5 122B-A10B fitted to one V100.1 result · 0 techniques · 1 parameter Configuration ID qwen35-llamab64739e-v100-fit1g-b128-c1k-f16 Created 2026-07-27T20:18:57Z Notes CUDA0 with automatic CPU fallback. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services stopped — The model router and ComfyUI were not resident. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 16 threads; 16 batch threads; NUMA interleave; CUDA0 with automatic fit; remaining tensors on CPU. Results and evidence `qwen35-v100-fit-20260727` Task text generation Status PROVISIONAL Measured 2026-07-27T20:18:57Z Statistic median; 5 samples Model Qwen3.5 122B-A10B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 1024 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 27.3 tok/s Decode 8.7 tok/s Evidence results/20260727-new-model-dual-v100/qwen35-single-gpu0.txt Benchmark notes. One-card control; correctness check pending. Result notes. Five-sample medians.

Qwen3.5 122B-A10B

Size: 125B / 10B
Quant: Q5_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	CPU + V100(2)	TODO	0	—	✕	—	—	—	15.6	NA	NA
	llama.cpp	CPU + V100(2)	TODO	0	—	✕	—	—	—	11.7	NA	NA
`q35q5c` Qwen3.5 Q5 fitted across two local V100s and host experts.2 results · 2 techniques · 15 parameters Configuration ID qwen35q5-llamab64739e-2v100-fit1g-layer-b1024-f16-copy Created 2026-08-30T06:11:26Z Notes The fit retained 25,554.57 MiB in CUDA_Host memory and assigned 18 layers to CUDA0 and 32 layers to CUDA1. Applied optimizations MoE expert placement The two-card fit retained 25,554.57 MiB in CUDA_Host; the three-card placement removed that host model buffer. Same-host layer split The two local V100s used a fitted layer split with host experts. Uncommon parameters accelerator.ecc enabled accelerator.gpu_clocks automatic accelerator.persistence_mode disabled backend.flash_attention requested; runtime reported disabled system.correctness_output_sha256 d7230984564c8927995c8d3d6725838080d1e4f8da951ddd5d328b7d823324be — Matched fixed-seed 64-token completion. system.cpu_governor schedutil system.page_cache_state not-applicable: no-mmap — The runtime allocated private model buffers. system.prefix_cache_state not-applicable — Zero-prompt llama-bench decode. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for each run. target.cuda_host_model_buffer 25554.57 MiB — Routed-expert tensors from all model blocks used CUDA_Host buffers. target.fit enabled target.fit_min_ctx 4096 tokens target.fit_target 1024 MiB per GPU target.model_buffers 500.16,29380.58,30794.45,25554.57 MiB — CPU, CUDA0, CUDA1, and CUDA_Host. target.tensor_split automatic; 18,32 layers — CUDA0 and CUDA1 shares from the final fitted load. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 18 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA0 share plus host expert execution. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 18 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; Repeated CUDA0 share plus host expert execution. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 32 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; CUDA1 share and main GPU. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 32 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; Repeated CUDA1 share and main GPU. Results and evidence `qwen35-q5-2v100-fit-control-20260830b` Task text generation Status INVALID Measured 2026-08-30T07:13:25Z Statistic median; 5 samples Model Qwen3.5 122B-A10B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 11.7 tok/s Evidence qwen35/results/20260830T071154Z-qwen35-q5-direct-rpc-3v100/two-v100-control.json Benchmark notes. Repeat samples: 12.2538, 11.6769, 11.8044, 10.2886, 9.78265 tok/s. Invalid for a speedup claim because the arm slowed across the run, spread 25.2605%, and did not reproduce the first median. Result notes. Retained as the invalid repeated CPU-offload control. `qwen35-q5-2v100-fit-control-20260830a` Task text generation Status INVALID Measured 2026-08-30T06:11:26Z Statistic median; 5 samples Model Qwen3.5 122B-A10B Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 15.6 tok/s Evidence qwen35/results/20260830T060951Z-qwen35-q5-direct-rpc-3v100/two-v100-control.json Benchmark notes. Zero-prompt decode samples: 14.5237, 15.8274, 16.6276, 15.6164, 11.3562 tok/s. Invalid for a speedup claim because the 46.4187% relative spread did not reproduce. Flash Attention was requested but the emitted benchmark JSON reports it disabled. Result notes. Retained as an invalid CPU-offload control.
	llama.cpp	V100(3)	TODO	0	—	✕	—	—	—	28.1	NA	NA
	llama.cpp	V100(3)	TODO	0	—	✕	—	—	—	27.6	NA	NA
`q35q53` Qwen3.5 Q5 split equally across two local and one remote V100.2 results · 3 techniques · 14 parameters Configuration ID qwen35q5-llamab64739e-3v100-rpc-equal-layer-b1024-f16-copy Created 2026-08-30T06:15:04Z Notes Fit target zero; 17:17:16 layers across CUDA0, CUDA1, and RPC0. No CUDA_Host model buffer remained. Applied optimizations Cross-host layer split Two local 17-layer shares and one remote 16-layer share used RPC TCP. GPU link topology RPC used the direct ConnectX-3 rail measured at 39.395 Gb/s. MoE expert placement The two-card fit retained 25,554.57 MiB in CUDA_Host; the three-card placement removed that host model buffer. Uncommon parameters accelerator.ecc enabled accelerator.gpu_clocks automatic accelerator.persistence_mode disabled backend.flash_attention requested; runtime reported disabled backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP iperf from the retained Laguna direct-link benchmark; 40 Gb/s link, MTU 9000, zero retransmits. system.correctness_output_sha256 d7230984564c8927995c8d3d6725838080d1e4f8da951ddd5d328b7d823324be — Matched fixed-seed 64-token completion. system.cpu_governor schedutil system.page_cache_state not-applicable: no-mmap — The runtime allocated private model buffers. system.prefix_cache_state not-applicable — Zero-prompt llama-bench decode. system.resident_gpu_services stopped — The R720 VRAM-consumer target was stopped for each run. target.fit disabled — llama-bench reported fit_target=0 and fit_min_ctx=0. target.model_buffers 500.16,29307.62,28968.24,27453.74 MiB — CPU, CUDA0, CUDA1, and RPC0; no CUDA_Host model buffer. target.tensor_split 1,1,1; 17,17,16 layers — CUDA0, CUDA1, and RPC0 shares from the load log. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 17 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 17 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Repeated local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 17 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share and main GPU. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 17 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Repeated local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 16 GPU layers; network enp65s0d1; RPC0 share on the remote V100. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 50 target layers; 16 GPU layers; network enp65s0d1; Repeated RPC0 share on the remote V100. Results and evidence `qwen35-q5-3v100-rpc-equal-20260830b` Task text generation Status VALID Measured 2026-08-30T07:17:20Z Statistic median; 5 samples Model Qwen3.5 122B-A10B Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 27.6 tok/s Baseline q35q5c Output matches baseline yes Evidence qwen35/results/20260830T071154Z-qwen35-q5-direct-rpc-3v100/three-v100-candidate.json Benchmark notes. Repeat samples: 27.5660, 26.6655, 28.0736, 29.4724, 27.4978 tok/s. Relative spread 10.5263%; its median reproduced the first three-card median within 1.9%. Fixed-seed output matched exactly. Result notes. Repeated the first candidate median within 1.9%. Exact median speedup withheld because both matched controls are invalid. `qwen35-q5-3v100-rpc-equal-20260830a` Task text generation Status VALID Measured 2026-08-30T06:15:04Z Statistic median; 5 samples Model Qwen3.5 122B-A10B Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Tested depth 0 Prompt tokens 0 Generated tokens 128 Decode 28.1 tok/s Baseline q35q5c Output matches baseline yes Evidence qwen35/results/20260830T060951Z-qwen35-q5-direct-rpc-3v100/three-v100-candidate.json Benchmark notes. Zero-prompt decode samples: 28.2743, 28.1666, 27.8451, 27.7790, 28.0916 tok/s. Relative spread 1.7830%. A fixed-seed completion matched the two-card output byte for byte. Result notes. Exact median speedup withheld because the matched control is invalid. Across both runs, candidate minimum divided by control maximum is 1.603689.

Step 3.5 Flash

Size: 197B / 11B
Quant: Q4_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(4)	32768	0	—	?	—	—	—	—	NA	NA
`s35eqf` Step 3.5 equal four-V100 placement at batch 1024 and 32K context.1 result · 5 techniques · 10 parameters Configuration ID step35-llama8172e657-4v100-rpc-equal-b1024-c32k-f16-fa Created 2026-09-13T17:11:43Z Notes Rejected capacity screen: RPC0 could not allocate the compute graph. Applied optimizations Batch-size tuning Batch and microbatch 1024 exhausted RPC0 during compute allocation. Cross-host layer split Two local and two remote V100 shares hold the complete model. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The retained target uses the Q4_K_M artifact. Uncommon parameters backend.failed_compute_allocation 644.14; retry 350.41 MiB — Both RPC0 allocations failed. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.model_cache_state warm — The mmap artifact had been read by verification and prior load attempts. system.prefix_cache_state not reached — Context creation failed before the server accepted a request. system.resident_gpu_services stopped — lab-gpu-vram-eaters.target was stopped and restored. target.fit disabled target.model_buffers 25275.26,29092.49,31606.04,28119.73 MiB — CUDA0, CUDA1, RPC0, and RPC1. target.tensor_split 1,1,1,1 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-equal-b1024-capacity-failure-20260913` Task text generation Status INVALID Measured 2026-09-13T17:11:43Z Statistic single; 1 sample Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 0 Prompt tokens 0 Generated tokens 0 Evidence step35/results/20260913T074415Z-step35-flash-q4-k-m-four-v100-day0/capacity-equal-split-b1024-failure Benchmark notes. The model loaded, but context creation failed before any request. sample_count 1 is the schema minimum; zero timed samples completed. Result notes. The equal placement consumed too much RPC0 memory even at 32K.
	llama.cpp	V100(4)	32768	24699	24699	?	—	—	245.7	27.3	4%	NA
`s3532h` Step 3.5 fitted four-V100 placement at 32K with a cache hit.1 result · 6 techniques · 12 parameters Configuration ID step35-llama8172e657-4v100-rpc-13111111-b512-c32k-f16-fa-cache-hit Created 2026-09-13T17:40:55Z Notes Restored 23,336 tokens and processed 1,363 tokens. Applied optimizations Batch-size tuning Reducing batch and microbatch from 1024 to 512 made the 128K full-GPU placement fit. Cross-host layer split Two local and two remote V100 shares hold the complete model. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching The miss and hit are retained separately; hit output divergence invalidates the serving result. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The retained target uses the Q4_K_M artifact. Uncommon parameters accelerator.local_free_vram 4429,2511 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.host_prompt_cache enabled MiB — --cache-ram 131072. system.model_cache_state warm — The mmap artifact had been read by verification and prior load attempts. system.prefix_cache_state hit cached tokens — Processed the remaining 1,363 prompt tokens. system.resident_gpu_services stopped — lab-gpu-vram-eaters.target was stopped and restored. target.fit disabled — The fitter output was copied into the queue as explicit arguments. target.model_buffers 26954.64,29112.56,28720.03,29306.29 MiB — CUDA0, CUDA1, RPC0, and RPC1; all 46 layers were offloaded. target.tensor_overrides blk.12.ffn_down=CUDA1; blk.23.ffn_down=RPC0; blk.34.ffn_down=RPC1 — Regex patterns are retained in each exact command file. target.tensor_split 13,11,11,11 — CUDA0, CUDA1, RPC0, and RPC1 order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 13 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-agent-32k-cache-hit-20260913` Task text generation Status INVALID Measured 2026-09-13T17:40:55Z Statistic median; 5 samples Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 24699 Prompt tokens 24699 Generated tokens 150 Prefill 245.7 tok/s Decode 27.3 tok/s Time to first token 5546.8 ms Baseline s3532m Output matches baseline no Evidence step35/results/20260913T074415Z-step35-flash-q4-k-m-four-v100-day0 Benchmark notes. Restored 23,336 tokens. The restored output reproduced across five samples but differed from the byte-identical cold request. Result notes. Prompt-time speedup is 11.916 times. Range 5,299.964 to 5,559.120 ms; first_divergent_token is one-based.
	llama.cpp	V100(4)	32768	24699	24699	?	—	—	373.7	26.8	7%	NA
`s3532m` Step 3.5 fitted four-V100 placement at 32K with a cache miss.1 result · 6 techniques · 12 parameters Configuration ID step35-llama8172e657-4v100-rpc-13111111-b512-c32k-f16-fa-cache-miss Created 2026-09-13T17:34:44Z Notes One slot, 128 GiB host prompt cache, and no speculative decoding. Applied optimizations Batch-size tuning Reducing batch and microbatch from 1024 to 512 made the 128K full-GPU placement fit. Cross-host layer split Two local and two remote V100 shares hold the complete model. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching The miss and hit are retained separately; hit output divergence invalidates the serving result. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The retained target uses the Q4_K_M artifact. Uncommon parameters accelerator.local_free_vram 4429,2511 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.host_prompt_cache enabled MiB — --cache-ram 131072. system.model_cache_state warm — The mmap artifact had been read by verification and prior load attempts. system.prefix_cache_state miss cached tokens — Processed all 24,699 prompt tokens. system.resident_gpu_services stopped — lab-gpu-vram-eaters.target was stopped and restored. target.fit disabled — The fitter output was copied into the queue as explicit arguments. target.model_buffers 26954.64,29112.56,28720.03,29306.29 MiB — CUDA0, CUDA1, RPC0, and RPC1; all 46 layers were offloaded. target.tensor_overrides blk.12.ffn_down=CUDA1; blk.23.ffn_down=RPC0; blk.34.ffn_down=RPC1 — Regex patterns are retained in each exact command file. target.tensor_split 13,11,11,11 — CUDA0, CUDA1, RPC0, and RPC1 order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 13 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-agent-32k-cache-miss-20260913` Task text generation Status VALID Measured 2026-09-13T17:34:44Z Statistic median; 5 samples Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 24699 Prompt tokens 24699 Generated tokens 150 Prefill 373.7 tok/s Decode 26.8 tok/s Time to first token 66096.6 ms Output matches baseline yes Evidence step35/results/20260913T074415Z-step35-flash-q4-k-m-four-v100-day0 Benchmark notes. Cold agent-style prefix. All five greedy token sequences matched. Result notes. Median prompt time. Range 65,667.379 to 66,258.672 ms.
	llama.cpp	V100(4)	32768	24699	24699	?	—	—	—	27.2	—	NA
`s35mtpb` Step 3.5 four-V100 32K decode control for the native-MTP screen.1 result · 4 techniques · 5 parameters Configuration ID step35-llama8172e657-4v100-rpc-13111111-b512-c32k-f16-mtp-control Created 2026-09-14T06:04:14Z Notes Three measured 400-token responses after one warm request. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold the complete target. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching Measured requests reused the prompt in the live server slot. Weight quantization The retained target uses Q4_K_M. Uncommon parameters accelerator.local_free_vram 4429,2511 MiB — R720 CUDA0 and CUDA1 after load. system.prefix_cache_state live slot hit cached tokens — One token was processed for each measured request. target.model_buffers 26954.64,29112.56,28720.03,29306.29 MiB — CUDA0, CUDA1, RPC0, and RPC1; all 46 layers were offloaded. target.tensor_overrides blk.12.ffn_down=CUDA1; blk.23.ffn_down=RPC0; blk.34.ffn_down=RPC1 — The retained command has the exact regular expressions. target.tensor_split 13,11,11,11 — CUDA0, CUDA1, RPC0, and RPC1 order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 13 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-mtp-screen-control-20260914` Task text generation Status VALID Measured 2026-09-14T06:04:14Z Statistic median; 3 samples Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 24699 Prompt tokens 24699 Generated tokens 400 Decode 27.2 tok/s Time to first token 240.1 ms Output matches baseline yes Evidence step35/results/20260914T060030Z-step35-mtp-screen Benchmark notes. Each request reused 24,698 tokens from the live server slot. All three generated token arrays matched. Result notes. Decode range 26.751093744280194 to 27.381887985980473 tok/s; spread was 2.358% of the minimum sample.
	llama.cpp	V100(4)	131072	109941	109941	?	—	—	155.1	23.5	3%	NA
`s35128h` Step 3.5 fitted four-V100 placement at 128K with a cache hit.1 result · 6 techniques · 12 parameters Configuration ID step35-llama8172e657-4v100-rpc-13111111-b512-c128k-f16-fa-cache-hit Created 2026-09-13T18:56:09Z Notes Restored 108,185 tokens and processed 1,756 tokens. Applied optimizations Batch-size tuning Reducing batch and microbatch from 1024 to 512 made the 128K full-GPU placement fit. Cross-host layer split Two local and two remote V100 shares hold the complete model. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching The miss and hit are retained separately; hit output divergence invalidates the serving result. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The retained target uses the Q4_K_M artifact. Uncommon parameters accelerator.local_free_vram 2797,1647 MiB — R720 CUDA0 and CUDA1 after load. Remote post-load free VRAM was not retained. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.host_prompt_cache enabled MiB — --cache-ram 131072. system.model_cache_state warm — The mmap artifact had been read by verification and prior load attempts. system.prefix_cache_state hit cached tokens — Processed the remaining 1,756 prompt tokens. system.resident_gpu_services stopped — lab-gpu-vram-eaters.target was stopped and restored. target.fit disabled — The fitter output was copied into the queue as explicit arguments. target.model_buffers 26954.64,29112.56,28720.03,29306.29 MiB — CUDA0, CUDA1, RPC0, and RPC1; all 46 layers were offloaded. target.tensor_overrides blk.12.ffn_down=CUDA1; blk.23.ffn_down=RPC0; blk.34.ffn_down=RPC1 — Regex patterns are retained in each exact command file. target.tensor_split 13,11,11,11 — CUDA0, CUDA1, RPC0, and RPC1 order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 13 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-agent-128k-cache-hit-20260913` Task text generation Status INVALID Measured 2026-09-13T18:56:09Z Statistic median; 5 samples Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 109941 Prompt tokens 109941 Generated tokens 150 Prefill 155.1 tok/s Decode 23.5 tok/s Time to first token 11324.1 ms Baseline s35128m Output matches baseline no Evidence step35/results/20260913T074415Z-step35-flash-q4-k-m-four-v100-day0 Benchmark notes. Restored 108,185 tokens. The restored output reproduced across five samples but differed from the byte-identical cold request. Result notes. Prompt-time speedup is 37.051 times. Range 10,739.735 to 11,583.914 ms; first_divergent_token is one-based.
	llama.cpp	V100(4)	131072	109941	109941	?	—	—	262.0	23.3	5%	NA
`s35128m` Step 3.5 fitted four-V100 placement at 128K with a cache miss.1 result · 6 techniques · 12 parameters Configuration ID step35-llama8172e657-4v100-rpc-13111111-b512-c128k-f16-fa-cache-miss Created 2026-09-13T18:44:12Z Notes One slot, 128 GiB host prompt cache, and no speculative decoding. Applied optimizations Batch-size tuning Reducing batch and microbatch from 1024 to 512 made the 128K full-GPU placement fit. Cross-host layer split Two local and two remote V100 shares hold the complete model. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching The miss and hit are retained separately; hit output divergence invalidates the serving result. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The retained target uses the Q4_K_M artifact. Uncommon parameters accelerator.local_free_vram 2797,1647 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.host_prompt_cache enabled MiB — --cache-ram 131072. system.model_cache_state warm — The mmap artifact had been read by verification and prior load attempts. system.prefix_cache_state miss cached tokens — Processed all 109,941 prompt tokens. system.resident_gpu_services stopped — lab-gpu-vram-eaters.target was stopped and restored. target.fit disabled — The fitter output was copied into the queue as explicit arguments. target.model_buffers 26954.64,29112.56,28720.03,29306.29 MiB — CUDA0, CUDA1, RPC0, and RPC1; all 46 layers were offloaded. target.tensor_overrides blk.12.ffn_down=CUDA1; blk.23.ffn_down=RPC0; blk.34.ffn_down=RPC1 — Regex patterns are retained in each exact command file. target.tensor_split 13,11,11,11 — CUDA0, CUDA1, RPC0, and RPC1 order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 13 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share and main GPU. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-agent-128k-cache-miss-20260913` Task text generation Status VALID Measured 2026-09-13T18:44:12Z Statistic median; 5 samples Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 109941 Prompt tokens 109941 Generated tokens 150 Prefill 262.0 tok/s Decode 23.3 tok/s Time to first token 419564.1 ms Output matches baseline yes Evidence step35/results/20260913T074415Z-step35-flash-q4-k-m-four-v100-day0 Benchmark notes. Cold agent-style prefix. All five greedy token sequences matched. Result notes. Median prompt time. Range 414,714.435 to 421,108.116 ms.
	llama.cpp	V100(4)	32768	0	—	Local CPU MTP	—	—	—	—	NA	NA
`s35mtp2` Step 3.5 four-V100 32K native-MTP width-two screen.1 result · 4 techniques · 5 parameters Configuration ID step35-llama8172e657-4v100-rpc-13111111-b512-c32k-f16-mtp2 Created 2026-09-14T06:15:57Z Notes Invalid: the artifact does not contain MTP layers. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold the complete target. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Speculative decoding Native MTP width two was requested but the artifact lacks MTP layers. Weight quantization The retained target uses Q4_K_M. Uncommon parameters draft.draft_width 2 tokens draft.mtp_head missing from artifact — llama.cpp rejected context type MTP after loading the target. target.model_buffers 26954.64,29112.56,28720.03,29306.29 MiB — CUDA0, CUDA1, RPC0, and RPC1; all 46 layers were offloaded. target.tensor_overrides blk.12.ffn_down=CUDA1; blk.23.ffn_down=RPC0; blk.34.ffn_down=RPC1 — The retained command has the exact regular expressions. target.tensor_split 13,11,11,11 — CUDA0, CUDA1, RPC0, and RPC1 order. Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 13 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 46 target layers; 11 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `step35-mtp2-missing-head-20260914` Task text generation Status INVALID Measured 2026-09-14T06:15:57Z Statistic single; 1 sample Model Step 3.5 Flash Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 0 Prompt tokens 0 Generated tokens 0 Evidence step35/results/20260914T060030Z-step35-mtp-screen/mtp2-server.log Benchmark notes. The complete target and normal 32K context loaded. MTP context creation then failed before the server accepted a request. sample_count 1 is the schema minimum; zero samples completed. Result notes. This is an artifact capability failure, not a VRAM failure.

MiniMax M2.5

Size: 229B / 10B
Quant: Q3_K_M

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	V100(4)	131072	0	—	?	—	—	—	—	NA	NA
`m25f16f` MiniMax M2.5 equal four-V100 placement with F16 KV at 128K.1 result · 4 techniques · 6 parameters Configuration ID minimax-m25-llama8172e657-4v100-rpc-equal-b1024-c128k-f16-failure Created 2026-09-14T04:32:30Z Notes Rejected capacity screen: CUDA0 could not allocate the KV buffer. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold all 63 offloaded layers. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Release GPU capacity Resident GPU consumers were stopped before the benchmark. Weight quantization The retained target uses Q3_K_M. Uncommon parameters backend.failed_kv_allocation 8192 MiB — CUDA0. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.resident_gpu_services stopped — GPU consumers were stopped for the run and left stopped overnight. target.tensor_split 1,1,1,1 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 15 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `minimax-m25-f16-kv-capacity-failure-20260914` Task text generation Status INVALID Measured 2026-09-14T04:32:30Z Statistic single; 1 sample Model MiniMax M2.5 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 0 Prompt tokens 0 Generated tokens 0 Evidence minimax-m25/results/20260914T031224Z-minimax-m25-q3-k-m-four-v100-day0/128k-miss-sample-1-load.log Benchmark notes. The model loaded, but context creation failed before any request. sample_count 1 is the schema minimum; zero timed samples completed. Result notes. The equal F16 placement exhausted memory at 128K.
	llama.cpp	V100(4)	131072	0	—	?	—	—	—	—	NA	NA
`m25q8rf` MiniMax M2.5 ratio-only four-V100 placement with Q8 KV at 128K.1 result · 5 techniques · 7 parameters Configuration ID minimax-m25-llama8172e657-4v100-rpc-ratio-b1024-c128k-q8-failure Created 2026-09-14T04:39:55Z Notes Rejected capacity screen: the ratio did not move layer boundaries. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold all 63 offloaded layers. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. KV-cache quantization Q8 K and V made the 128K KV allocation fit. Release GPU capacity Resident GPU consumers were stopped before the benchmark. Weight quantization The retained target uses Q3_K_M. Uncommon parameters backend.failed_compute_allocation 1656.39; retry 840.10 MiB — Both CUDA0 allocations failed. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.resident_gpu_services stopped — GPU consumers were stopped for the run and left stopped overnight. target.model_buffers 27022.95,26590.20,26590.20,23747.25 MiB — The ratio did not change the discrete layer boundaries. target.tensor_split 15,16,16,15 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 15 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `minimax-m25-q8-compute-capacity-failure-20260914` Task text generation Status INVALID Measured 2026-09-14T04:39:55Z Statistic single; 1 sample Model MiniMax M2.5 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 0 Prompt tokens 0 Generated tokens 0 Evidence minimax-m25/results/20260914T031224Z-minimax-m25-q3-k-m-four-v100-day0/128k-miss-sample-1-load.log Benchmark notes. Q8 KV fit, but ratio-only placement exhausted CUDA0 compute memory. Result notes. Q8 KV fit, but the ratio alone did not move a layer boundary.
	llama.cpp	V100(4)	32768	23827	23827	?	—	—	230.5	20.9	4%	NA
`m2532h` MiniMax M2.5 equal four-V100 placement at 32K with a cache hit.1 result · 5 techniques · 8 parameters Configuration ID minimax-m25-llama8172e657-4v100-rpc-equal-b1024-c32k-f16-cache-hit Created 2026-09-14T03:29:42Z Notes Restored 22,954 tokens and processed 873 tokens. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold all 63 offloaded layers. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching The miss and hit are separate; output divergence invalidates the hit. Release GPU capacity Resident GPU consumers were stopped before the benchmark. Weight quantization The retained target uses Q3_K_M. Uncommon parameters accelerator.local_free_vram 2555,2987 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.prefix_cache_state hit cached tokens system.resident_gpu_services stopped — GPU consumers were stopped for the run and left stopped overnight. target.model_buffers 27022.95,26590.20,26590.20,23747.25 MiB — CUDA0, CUDA1, RPC0, and RPC1. target.tensor_split 1,1,1,1 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 15 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `minimax-m25-agent-32k-cache-hit-20260914` Task text generation Status INVALID Measured 2026-09-14T03:29:42Z Statistic median; 5 samples Model MiniMax M2.5 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 23827 Prompt tokens 23827 Generated tokens 150 Prefill 230.5 tok/s Decode 20.9 tok/s Time to first token 3786.8 ms Baseline m2532m Output matches baseline no Evidence minimax-m25/results/20260914T031224Z-minimax-m25-q3-k-m-four-v100-day0 Benchmark notes. The restored output reproduced across five samples but differed from the byte-identical cold request. Result notes. Prompt-time ratio is 17.869. Range 3,772.127 to 3,881.934 ms; first_divergent_token is one-based.
	llama.cpp	V100(4)	32768	23827	23827	?	—	—	352.1	21.3	6%	NA
`m2532m` MiniMax M2.5 equal four-V100 placement at 32K with a cache miss.1 result · 5 techniques · 8 parameters Configuration ID minimax-m25-llama8172e657-4v100-rpc-equal-b1024-c32k-f16-cache-miss Created 2026-09-14T03:19:47Z Notes One slot, 128 GiB host prompt cache, and no speculative decoding. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold all 63 offloaded layers. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. Prefix caching The miss and hit are separate; output divergence invalidates the hit. Release GPU capacity Resident GPU consumers were stopped before the benchmark. Weight quantization The retained target uses Q3_K_M. Uncommon parameters accelerator.local_free_vram 2555,2987 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.prefix_cache_state miss cached tokens system.resident_gpu_services stopped — GPU consumers were stopped for the run and left stopped overnight. target.model_buffers 27022.95,26590.20,26590.20,23747.25 MiB — CUDA0, CUDA1, RPC0, and RPC1. target.tensor_split 1,1,1,1 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 15 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `minimax-m25-agent-32k-cache-miss-20260914` Task text generation Status VALID Measured 2026-09-14T03:19:47Z Statistic median; 5 samples Model MiniMax M2.5 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 32768 Tested depth 23827 Prompt tokens 23827 Generated tokens 150 Prefill 352.1 tok/s Decode 21.3 tok/s Time to first token 67664.5 ms Output matches baseline yes Evidence minimax-m25/results/20260914T031224Z-minimax-m25-q3-k-m-four-v100-day0 Benchmark notes. Cold agent-style prefix. All five greedy token sequences matched. Result notes. Median prompt time. Range 67,058.077 to 69,201.587 ms.
	llama.cpp	V100(4)	131072	104561	104561	?	—	—	100.2	6.9	2%	NA
`m25128h` MiniMax M2.5 fitted four-V100 placement at 128K with a cache hit.1 result · 6 techniques · 9 parameters Configuration ID minimax-m25-llama8172e657-4v100-rpc-fitted-b1024-c128k-q8-cache-hit Created 2026-09-14T05:01:23Z Notes Restored 103,428 tokens and processed 1,133 tokens. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold all 63 offloaded layers. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. KV-cache quantization Q8 K and V made the 128K KV allocation fit. Prefix caching The miss and hit are separate; output divergence invalidates the hit. Release GPU capacity Resident GPU consumers were stopped before the benchmark. Weight quantization The retained target uses Q3_K_M. Uncommon parameters accelerator.local_free_vram 1617,363 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.prefix_cache_state hit cached tokens system.resident_gpu_services stopped — GPU consumers were stopped for the run and left stopped overnight. target.model_buffers 25361.07,26590.20,26590.20,25409.13 MiB — CUDA0, CUDA1, RPC0, and RPC1. target.tensor_overrides blk.15=CUDA1; blk.31=RPC0; blk.47=RPC1 — The queue retains the exact regular expressions. target.tensor_split 15,16,16,15 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 15 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `minimax-m25-agent-128k-cache-hit-20260914` Task text generation Status INVALID Measured 2026-09-14T05:01:23Z Statistic single; 1 sample Model MiniMax M2.5 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 104561 Prompt tokens 104561 Generated tokens 150 Prefill 100.2 tok/s Decode 6.9 tok/s Time to first token 11304.0 ms Baseline m25128m Output matches baseline no Evidence minimax-m25/results/20260914T031224Z-minimax-m25-q3-k-m-four-v100-day0 Benchmark notes. One restored-prefix sample differed from the cold output. Result notes. Prompt-time ratio is 50.323; first_divergent_token is one-based.
	llama.cpp	V100(4)	131072	104561	104561	?	—	—	183.8	7.2	3%	NA
`m25128m` MiniMax M2.5 fitted four-V100 placement at 128K with a cache miss.1 result · 6 techniques · 9 parameters Configuration ID minimax-m25-llama8172e657-4v100-rpc-fitted-b1024-c128k-q8-cache-miss Created 2026-09-14T04:46:53Z Notes Q8 KV and three explicit layer-boundary overrides. Applied optimizations Cross-host layer split Two local and two remote V100 shares hold all 63 offloaded layers. GPU link topology RPC used the direct 40 GbE ConnectX-3 rail. KV-cache quantization Q8 K and V made the 128K KV allocation fit. Prefix caching The miss and hit are separate; output divergence invalidates the hit. Release GPU capacity Resident GPU consumers were stopped before the benchmark. Weight quantization The retained target uses Q3_K_M. Uncommon parameters accelerator.local_free_vram 1617,363 MiB — R720 CUDA0 and CUDA1 after load. backend.load_mode mmap backend.rpc_transport TCP — llama.cpp RPC on 10.33.3.2:50052; no NCCL or verbs. network.direct_link 10.33.3.1 to 10.33.3.2 Mbit/second — Four-stream TCP measurement retained with the Laguna RPC benchmark. system.prefix_cache_state miss cached tokens system.resident_gpu_services stopped — GPU consumers were stopped for the run and left stopped overnight. target.model_buffers 25361.07,26590.20,26590.20,25409.13 MiB — CUDA0, CUDA1, RPC0, and RPC1. target.tensor_overrides blk.15=CUDA1; blk.31=RPC0; blk.47=RPC1 — The queue retains the exact regular expressions. target.tensor_split 15,16,16,15 Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 15 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA0 share. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; 16 threads; 16 batch threads; affinity ffff; NUMA interleave; network roce0; Local CUDA1 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC0 share. c720 — target; CUDA; Tesla PG500-216 (V100 32GB); 63 target layers; 16 GPU layers; network enp65s0d1; Remote RPC1 share. Results and evidence `minimax-m25-agent-128k-cache-miss-20260914` Task text generation Status PROVISIONAL Measured 2026-09-14T04:46:53Z Statistic single; 1 sample Model MiniMax M2.5 Backend llama.cpp Hosts r720,c720 Hardware Tesla PG500-216 (V100 32GB) Context window 131072 Tested depth 104561 Prompt tokens 104561 Generated tokens 150 Prefill 183.8 tok/s Decode 7.2 tok/s Time to first token 568850.8 ms Output matches baseline yes Evidence minimax-m25/results/20260914T031224Z-minimax-m25-q3-k-m-four-v100-day0 Benchmark notes. Cold agent-style prefix. One complete sample was retained. Result notes. One complete cold sample; retain as provisional.

DeepSeek V4 Flash 0731

Size: 284B / 13B
Quant: UD-Q8_K_XL

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	llama.cpp	CPU + V100(2)	4096	512	—	✕	—	—	15.9	4.1	NA	NA
`d4bal` DeepSeek V4 Flash balanced optimized profile.1 result · 4 techniques · 5 parameters Configuration ID deepseek4-b10216-2v100-ncmoe24-fit1536-t20-b128 Created 2026-08-01T23:25:29Z Notes Recommended profile; retained prompt speed and improved decode. Applied optimizations Batch-size tuning Batch 128 preserved prompt throughput and matched the retained decode rate of batch 64. CPU thread tuning Twenty physical threads beat 12 and 16 in the short sweep. GPU layer offload A 1,536 MiB per-GPU reserve won the fit-margin sweep. MoE expert placement CPU expert cutoff 24 won a 0, 22, 24, 26, 28, and 30 sweep. Uncommon parameters system.gpu_services unloaded — ComfyUI and the router released GPU model memory. system.gpu_thermal_control active — The thermal controller stayed active during every measurement. system.host_weight_repack disabled — The benchmark-only runtime disabled the extra CPU weight copy. target.fit_target 1536 MiB per GPU target.n_cpu_moe 24 layers Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 43 target layers; 20 threads; 20 batch threads; NUMA interleave; CUDA0 and interleaved host expert weights. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 43 target layers; 20 threads; 20 batch threads; NUMA interleave; CUDA1 layer split and main GPU. Results and evidence `deepseek4-balanced-optimized-20260801` Task text generation Status VALID Measured 2026-08-01T23:25:29Z Statistic mean; 5 samples Model DeepSeek V4 Flash 0731 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 15.9 tok/s Decode 4.1 tok/s Baseline d4base (+14.3%) Evidence deepseek4/results/20260801T232529Z-final/README.md Benchmark notes. Balanced recommended profile. Result notes. Decode improved 14.3%; prompt throughput changed by less than 0.1%. Decode samples: 4.01034, 4.35316, 4.00644, 4.27133, 3.87861.
	llama.cpp	CPU + V100(2)	4096	512	—	✕	—	—	15.9	3.6	NA	NA
`d4base` DeepSeek V4 Flash baseline with automatic fit on two V100s.1 result · 0 techniques · 5 parameters Configuration ID deepseek4-b10216-2v100-auto-fit2048-t16-b128 Created 2026-08-01T20:51:00Z Notes Equal layer split, main GPU 1, F16 KV, Flash Attention. Applied optimizations No applied technique is cataloged. Uncommon parameters system.gpu_services unloaded — ComfyUI and the router released GPU model memory. system.gpu_thermal_control active — The thermal controller stayed active during every measurement. system.host_weight_repack disabled — The benchmark-only runtime disabled the extra CPU weight copy. target.fit_target 2048 MiB per GPU target.n_cpu_moe 0 layers — No explicit CPU expert cutoff; automatic fit controlled fallback. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 43 target layers; 16 threads; 16 batch threads; NUMA interleave; CUDA0 and interleaved host expert weights. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 43 target layers; 16 threads; 16 batch threads; NUMA interleave; CUDA1 layer split and main GPU. Results and evidence `deepseek4-baseline-20260801` Task text generation Status VALID Measured 2026-08-01T20:51:00Z Statistic mean; 5 samples Model DeepSeek V4 Flash 0731 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 15.9 tok/s Decode 3.6 tok/s Evidence deepseek4/results/20260801T205100Z-final/README.md Benchmark notes. Retained automatic-fit baseline. Result notes. Prompt samples: 16.1363, 16.1980, 15.8813, 15.5785, 15.6728. Decode samples: 3.63349, 3.40931, 3.67549, 3.63351, 3.59987.
	llama.cpp	CPU + V100(2)	4096	512	—	✕	—	—	10.6	4.1	NA	NA
`d4dec` DeepSeek V4 Flash decode-focused optimized profile.1 result · 1 technique · 5 parameters Configuration ID deepseek4-b10216-2v100-ncmoe24-fit1536-t20-b64 Created 2026-08-01T23:11:23Z Notes Batch 64 increased decode but reduced prompt throughput. Applied optimizations Batch-size tuning Batch 64 gave stable decode but reduced prompt throughput by 33.6%. Uncommon parameters system.gpu_services unloaded — ComfyUI and the router released GPU model memory. system.gpu_thermal_control active — The thermal controller stayed active during every measurement. system.host_weight_repack disabled — The benchmark-only runtime disabled the extra CPU weight copy. target.fit_target 1536 MiB per GPU target.n_cpu_moe 24 layers Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 43 target layers; 20 threads; 20 batch threads; NUMA interleave; CUDA0 and interleaved host expert weights. r720 — target; CUDA; Tesla PG500-216 (V100 32GB); 43 target layers; 20 threads; 20 batch threads; NUMA interleave; CUDA1 layer split and main GPU. Results and evidence `deepseek4-decode-optimized-20260801` Task text generation Status VALID Measured 2026-08-01T23:11:23Z Statistic mean; 5 samples Model DeepSeek V4 Flash 0731 Backend llama.cpp Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 512 Generated tokens 128 Prefill 10.6 tok/s Decode 4.1 tok/s Baseline d4base (+13.8%) Evidence deepseek4/results/20260801T231123Z-final/README.md Benchmark notes. Decode-focused profile; prompt throughput regression retained. Result notes. Decode improved 13.8%; prompt throughput fell 33.6%. Decode samples: 3.95639, 4.17102, 4.05723, 4.12847, 4.11271.

GLM-5.2

Size: 744B / 40B
Quant: INT4 g64 + INT8 MTP

Cfg	Backend	Hardware	Window	Prompt	Test depth	Speculative			Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
Cfg	Backend	Hardware	Window	Prompt	Test depth	Enabled	Speedup	Accept	Prefill tok/s	Decode tok/s	Prefill peak	Decode peak
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.6	NA	NA
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.8	NA	NA
`col2g` Colibri GLM-5.2 with both V100s, buffered async I/O, and CUDA dense placement.2 results · 5 techniques · 13 parameters Configuration ID glm52-colibri-2v100-r278-v58-pipe-dense Created 2026-08-06T05:53:28Z Notes Selected service profile: COLI_GPUS=0,1, CUDA_EXPERT_GB=58, CUDA_RELEASE_HOST=1, PIPE=1, CUDA_DENSE=1, DIRECT=0, CACHE_ROUTE=0, and MTP=0. Applied optimizations Asynchronous expert I/O PIPE=1 hid all but 1.86% of the warmed median decode window. PIPE=1 overlapped expert reads with resident matmul. GPU layer offload CUDA_DENSE=1 placed dense tensors across both V100s. CUDA_DENSE=1 placed dense tensors across the selected CUDA devices. MoE expert placement Both V100s held 2,468 usage-ranked experts in 52.40 GB total. Both V100s held a shared 58 GB usage-ranked expert tier. NUMA memory policy COLI_NUMA=1 interleaved host expert slabs across both sockets. Release GPU capacity The router and ComfyUI released GPU capacity and were restored. Uncommon parameters backend.cache_route disabled backend.cuda_dense enabled — CUDA_DENSE=1. backend.cuda_devices 0,1 devices — COLI_GPUS=0,1. backend.cuda_expert_gb 58 GB — Total budget across both V100s. backend.cuda_release_host enabled — GPU-resident expert host copies were released. backend.direct_io disabled backend.draft disabled — DRAFT=0 and MTP=0. backend.pipe enabled — PIPE=1 with eight workers. backend.ram_gb 278 GiB request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.page_cache warm-not-dropped — One untimed warm request preceded five retained samples. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before each run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 20 threads; NUMA interleave; GPU 0 held 1,445 usage-ranked experts in 30.68 GB. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 20 threads; NUMA interleave; GPU 0 held usage-ranked experts. Host expert slabs were interleaved across both CPU sockets. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 20 threads; NUMA interleave; GPU 1 held 1,023 usage-ranked experts in 21.72 GB plus selected dense tensors. GPU expert host copies were released and the dynamic LRU remained in host RAM. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 20 threads; NUMA interleave; GPU 1 held usage-ranked experts and the selected dense tensors. Results and evidence `glm52-colibri-2v100-confirm-20260806` Task text generation Status VALID Measured 2026-08-06T06:08:58Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 128 Decode 0.6 tok/s Elapsed 232.492 s Peak VRAM 60.8 GiB Baseline colq1 (+257.9%) Output matches baseline yes Evidence colibri/results/20260806T060858Z-glm52-v100-dual-confirm/README.md Benchmark notes. Five retained warm samples. All text and canonical token sequences matched each other and the original quality baseline exactly. Result notes. Decode samples were 0.416817274471, 0.516950316228, 0.555348264103, 0.550556578291, and 0.554203721824 tokens/s. Elapsed time is the median-throughput sample profile window. Peak VRAM is the sum of the final per-GPU nvidia-smi snapshots, not a continuous monitor. `glm52-colibri-2v100-screen-20260806` Task text generation Status PROVISIONAL Measured 2026-08-06T05:53:28Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.8 tok/s Elapsed 42.558 s Baseline colpg (+20.8%) Output matches baseline yes Evidence colibri/results/20260806T055328Z-glm52-v100-dual-screen/README.md Benchmark notes. Short dual-V100 validation screen. The output matched the retained quality baseline exactly. Result notes. One 32-token dual-V100 validation sample after one warm request.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.1	NA	NA
`cg4g32` GLM-5.2 with generic grouped-int4 CUDA kernels on two V100s.1 result · 5 techniques · 14 parameters Configuration ID glm52-colibri-2v100-sse41-g4generic32 Created 2026-08-15T00:27:51Z Notes Same G64-warp closure as cg4w32, with the runtime fallback selected. Applied optimizations Architecture-specific build The engine kept generic x86-64 with targeted SSE4.1 CPU functions; CUDA used sm_70. MoE expert placement Both V100s held a shared 58 GB usage-ranked expert tier. NUMA memory policy COLI_NUMA=1 interleaved host expert slabs across both sockets. Release GPU capacity Resident GPU services were released before retained measurements. Weight quantization The target artifact uses grouped-int4 expert weights with f32 scales. Uncommon parameters backend.cache_route disabled backend.cuda_dense disabled — CUDA_DENSE=0. backend.cuda_devices 0,1 devices backend.cuda_expert_gb 58 GB — Total across both V100s. backend.direct_io disabled — DIRECT=0. backend.draft disabled — DRAFT=0 and MTP=0. backend.grouped_g64_cuda_kernel generic — COLI_CUDA_G4_G64_WARP=0. backend.omp_threads 40 threads — OMP_DYNAMIC=FALSE. backend.pipe disabled — PIPE=0. request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.gpu_service_lifecycle target stopped; per-request guards — All three retained samples passed. system.page_cache dropped-before-load; one untimed warm request target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held 1,366 usage-ranked experts. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held 1,365 usage-ranked experts. Results and evidence `glm52-colibri-g4generic-confirm-20260815` Task text generation Status VALID Measured 2026-08-15T00:27:51Z Statistic median; 3 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.1 tok/s Elapsed 215.625 s Evidence colibri/results/20260815T002751Z-glm52-v100-g64-generic/README.md Benchmark notes. The same new closure used COLI_CUDA_G4_G64_WARP=0. Greedy decoding produced exact text and canonical tokens in all samples. Result notes. Decode samples were 0.148405797101, 0.143595634693, and 0.148786691030 tokens/s. Every sample reported zero expert disk service and zero felt read wait.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.1	NA	NA
`cg4w32` GLM-5.2 with warp-per-G64 CUDA kernels on two V100s.1 result · 6 techniques · 14 parameters Configuration ID glm52-colibri-2v100-sse41-g4warp32 Created 2026-08-15T01:02:57Z Notes Matched to cg4g32 except for COLI_CUDA_G4_G64_WARP=1. Applied optimizations Architecture-specific build The engine kept generic x86-64 with targeted SSE4.1 CPU functions; CUDA used sm_70. Kernel-path tuning Homogeneous format-4 G64 expert groups used the warp-per-group path. MoE expert placement Both V100s held a shared 58 GB usage-ranked expert tier. NUMA memory policy COLI_NUMA=1 interleaved host expert slabs across both sockets. Release GPU capacity Resident GPU services were released before retained measurements. Weight quantization The target artifact uses grouped-int4 expert weights with f32 scales. Uncommon parameters backend.cache_route disabled backend.cuda_dense disabled — CUDA_DENSE=0. backend.cuda_devices 0,1 devices backend.cuda_expert_gb 58 GB — Total across both V100s. backend.direct_io disabled — DIRECT=0. backend.draft disabled — DRAFT=0 and MTP=0. backend.grouped_g64_cuda_kernel warp-per-group — COLI_CUDA_G4_G64_WARP=1. backend.omp_threads 40 threads — OMP_DYNAMIC=FALSE. backend.pipe disabled — PIPE=0. request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.gpu_service_lifecycle target stopped; per-request guards — All three retained samples passed. system.page_cache dropped-before-load; one untimed warm request target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held 1,366 usage-ranked experts. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held 1,365 usage-ranked experts. Results and evidence `glm52-colibri-g4warp-confirm-20260815` Task text generation Status VALID Measured 2026-08-15T01:02:57Z Statistic median; 3 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.1 tok/s Elapsed 216.119 s Baseline cg4g32 (-0.2%) Output matches baseline yes Evidence colibri/results/20260815T010257Z-glm52-v100-g64-warp/README.md Benchmark notes. COLI_CUDA_G4_G64_WARP=1 was the only matched kernel change. Greedy output matched the generic control exactly. Result notes. Decode samples were 0.148341128969, 0.148066574434, and 0.147498064088 tokens/s. The 0.23% reduction is inside run noise. Candidate and control text SHA-256 was 481c7b79d21c977626bd72488b5189bf6818ade1782a9b36280b48b5f0913bb6; their canonical token SHA-256 was e6b16a313a2701b581aa4efbf35b2fcc9bd30ed80a8ffe3f174c1f6275a52111.
	Colibri	CPU + V100	4096	17	—	—	—	—	—	0.6	NA	NA
`colpg` Colibri GLM-5.2 with async buffered I/O and CUDA dense placement.1 result · 2 techniques · 11 parameters Configuration ID glm52-colibri-v100-r278-v29-pipe-dense Created 2026-08-06T00:55:32Z Notes Selected service profile: PIPE=1, CUDA_DENSE=1, DIRECT=0, CACHE_ROUTE=0, and MTP=0. Applied optimizations Asynchronous expert I/O PIPE=1 remained enabled. GPU layer offload CUDA_DENSE=1 placed non-expert tensors on GPU 0. Uncommon parameters backend.cache_route disabled backend.cuda_dense enabled — CUDA_DENSE=1. backend.cuda_expert_gb 29 GB — Dense tensors clamp the realizable expert tier. backend.direct_io disabled backend.draft disabled — DRAFT=0 and MTP=0. backend.pipe enabled — PIPE=1 with eight workers. backend.ram_gb 278 GiB request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.page_cache warm-not-dropped — One untimed warm request preceded retained samples. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before each run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; One V100 held resident dense tensors and a clamped hot-expert tier. Results and evidence `glm52-colibri-pipe-dense-screen-20260806` Task text generation Status PROVISIONAL Measured 2026-08-06T00:55:32Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.6 tok/s Elapsed 51.400 s Baseline colq1 Output matches baseline yes Evidence colibri/results/20260806T005532Z-glm52-v100-screen-pipe-dense Benchmark notes. Buffered direct-I/O control and selected profile. The short output matched the baseline prefix. Result notes. One 32-token buffered-I/O control.
	Colibri	CPU + V100	4096	17	—	—	—	—	—	0.6	NA	NA
`colpdd` Colibri GLM-5.2 with async direct I/O and CUDA dense placement.1 result · 3 techniques · 3 parameters Configuration ID glm52-colibri-v100-r278-v29-pipe-direct-dense Created 2026-08-06T00:15:06Z Notes Matched to colpd except for CUDA_DENSE=1. Applied optimizations Asynchronous expert I/O PIPE=1 remained enabled. Direct storage I/O DIRECT=1 remained enabled. GPU layer offload CUDA_DENSE=1 placed non-expert tensors on GPU 0. Uncommon parameters backend.cuda_dense enabled — CUDA_DENSE=1. backend.direct_io enabled — DIRECT=1. backend.pipe enabled — PIPE=1 with eight workers. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; One V100 held resident dense tensors and a clamped hot-expert tier. Results and evidence `glm52-colibri-pipe-direct-dense-screen-20260806` Task text generation Status PROVISIONAL Measured 2026-08-06T00:15:06Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.6 tok/s Elapsed 51.437 s Baseline colq1 Output matches baseline yes Evidence colibri/results/20260806T001506Z-glm52-v100-screen-pipe-direct-dense Benchmark notes. Short CUDA-dense screen; output matched the baseline prefix. Result notes. One 32-token CUDA-dense screen.
	Colibri	CPU + V100	4096	17	—	—	—	—	—	0.4	NA	NA
`colpd` Colibri GLM-5.2 with async direct expert I/O and CPU dense tensors.1 result · 2 techniques · 3 parameters Configuration ID glm52-colibri-v100-r278-v29-pipe-direct Created 2026-08-05T23:58:04Z Notes Matched to colp except for DIRECT=1. Applied optimizations Asynchronous expert I/O PIPE=1 remained enabled. Direct storage I/O DIRECT=1 was tested on the mixed-disk ZFS model volume. Uncommon parameters backend.cuda_dense disabled backend.direct_io enabled — DIRECT=1. backend.pipe enabled — PIPE=1 with eight workers. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; One V100 held routed experts; dense tensors remained on CPU. Results and evidence `glm52-colibri-pipe-direct-screen-20260805` Task text generation Status PROVISIONAL Measured 2026-08-05T23:58:04Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.4 tok/s Elapsed 81.870 s Baseline colq1 Output matches baseline yes Evidence colibri/results/20260805T235804Z-glm52-v100-screen-pipe-direct Benchmark notes. Short matched direct-I/O screen; output matched the baseline prefix. Result notes. One 32-token direct-I/O screen.
	Colibri	CPU + V100	4096	17	—	—	—	—	—	0.4	NA	NA
`colp` Colibri GLM-5.2 with async expert I/O and CPU-resident dense tensors.1 result · 1 technique · 3 parameters Configuration ID glm52-colibri-v100-r278-v29-pipe Created 2026-08-05T23:40:37Z Notes Quality policy, one V100, one serial client, one KV slot, fixed usage history, greedy decode, and a 278 GiB RAM budget. Applied optimizations Asynchronous expert I/O PIPE=1 overlapped missing-expert reads with resident matmul. Uncommon parameters backend.cuda_dense disabled backend.direct_io disabled backend.pipe enabled — PIPE=1 with eight workers. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; One V100 held routed experts; dense tensors remained on CPU. Results and evidence `glm52-colibri-pipe-screen-20260805` Task text generation Status PROVISIONAL Measured 2026-08-05T23:40:37Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.4 tok/s Elapsed 82.971 s Baseline colq1 Output matches baseline yes Evidence colibri/results/20260805T234037Z-glm52-v100-screen-pipe Benchmark notes. Short matched screen after one untimed warm request. The 32-token output matched the baseline prefix exactly. Result notes. One 32-token screen after one warm request.
	Colibri	CPU + V100	4096	17	—	—	—	—	—	0.3	NA	NA
`colr` Colibri balanced cache routing on the direct-I/O CUDA-dense profile.1 result · 1 technique · 4 parameters Configuration ID glm52-colibri-v100-r278-v29-pipe-direct-dense-route3x16 Created 2026-08-06T00:27:02Z Notes CACHE_ROUTE=1, ROUTE_J=3, ROUTE_M=16, and ROUTE_ALPHA=0.5. Applied optimizations Cache-aware expert routing Balanced max-rank routing used J=3, M=16, and alpha=0.5. Uncommon parameters backend.cache_route enabled — ROUTE_J=3, ROUTE_M=16, ROUTE_ALPHA=0.5. backend.cuda_dense enabled backend.direct_io enabled backend.pipe enabled Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; One V100 held resident dense tensors and hot routed experts. Results and evidence `glm52-colibri-route-screen-20260806` Task text generation Status PROVISIONAL Measured 2026-08-06T00:27:02Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.3 tok/s Elapsed 123.934 s Baseline colpdd Output matches baseline no Evidence colibri/results/20260806T002702Z-glm52-v100-screen-balanced-route Benchmark notes. Rejected short screen. Cache-aware routing changed the token sequence and reduced throughput. Result notes. The fourth generated token was the first divergence from the baseline prefix. Final reported expert hit rate was 96.4%.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.2	NA	NA
`cbs40` Provisional Colibri GLM-5.2 scalar W4A32 control with two V100s.1 result · 6 techniques · 16 parameters Configuration ID glm52-colibri-2v100-scalar-omp40-coldwarm32 Created 2026-08-14T22:23:07Z Notes Host caches were released before load. One untimed 32-token warm request preceded five retained 32-token samples. The GPU-service target restarted after sample 1, so this configuration is not the authoritative matched control. Applied optimizations Asynchronous expert I/O PIPE=1 used eight workers for missing-expert service. GPU layer offload CUDA_DENSE=1 placed selected dense tensors on the two V100s. MoE expert placement Both V100s held a shared 58 GB usage-ranked expert tier. NUMA memory policy COLI_NUMA=1 interleaved host expert slabs across both sockets. Release GPU capacity Resident GPU services were released before the retained region. Weight quantization The target artifact used grouped-int4 expert weights with f32 scales. Uncommon parameters backend.cache_route disabled backend.cuda_dense enabled — CUDA_DENSE=1. backend.cuda_devices 0,1 devices — COLI_GPUS=0,1. backend.cuda_expert_gb 58 GB — Total budget across both V100s. backend.cuda_pipeline enabled — COLI_CUDA_PIPE=1. backend.cuda_release_host enabled backend.direct_io disabled — DIRECT=0. backend.draft disabled — DRAFT=0 and MTP=0. backend.grouped_w4a32_kernel scalar backend.omp_threads 40 threads — OMP_DYNAMIC=FALSE. backend.pipe enabled — PIPE=1 with eight workers. backend.ram_gb 278 GiB request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.page_cache dropped-before-load; one untimed warm request — sudo sync and vm.drop_caches=3 preceded model load. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. Host expert slabs were interleaved. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts and selected dense tensors. Results and evidence `glm52-colibri-scalar-omp40-confirm-20260814` Task text generation Status PROVISIONAL Measured 2026-08-14T22:23:07Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.2 tok/s Elapsed 186.916 s Evidence colibri/results/20260814T223000Z-glm52-v100-matched-scalar-t40-confirm/README.md Benchmark notes. Pinned scalar package control. The router and ComfyUI restarted after sample 1, so the run is retained as provisional evidence. Colibri does not accept a per-request seed; temperature zero made decoding greedy. Result notes. Decode samples were 0.171199897280, 0.168864544250, 0.170375891811, 0.172353432257, and 0.172581167080 tokens/s. Elapsed time is the median-throughput sample profile window. All five scalar responses had identical text and canonical tokens.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.2	NA	NA
`cbx40` Colibri GLM-5.2 scalar W4A32 control with guarded GPU-service release.1 result · 6 techniques · 17 parameters Configuration ID glm52-colibri-2v100-scalar-omp40-exclusive32 Created 2026-08-14T23:14:21Z Notes Five valid samples from two matched invocations: four passed target and consumer guards before and after each request; one completed before an independently recorded service restart. One contaminated sample was rejected. Applied optimizations Asynchronous expert I/O PIPE=1 used eight workers for missing-expert service. GPU layer offload CUDA_DENSE=1 placed selected dense tensors on the two V100s. MoE expert placement Both V100s held a shared 58 GB usage-ranked expert tier. NUMA memory policy COLI_NUMA=1 interleaved host expert slabs across both sockets. Release GPU capacity Resident GPU services were released before the retained region. Weight quantization The target artifact used grouped-int4 expert weights with f32 scales. Uncommon parameters backend.cache_route disabled backend.cuda_dense enabled — CUDA_DENSE=1. backend.cuda_devices 0,1 devices — COLI_GPUS=0,1. backend.cuda_expert_gb 58 GB — Total budget across both V100s. backend.cuda_pipeline enabled — COLI_CUDA_PIPE=1. backend.cuda_release_host enabled backend.direct_io disabled — DIRECT=0. backend.draft disabled — DRAFT=0 and MTP=0. backend.grouped_w4a32_kernel scalar backend.omp_threads 40 threads — OMP_DYNAMIC=FALSE. backend.pipe enabled — PIPE=1 with eight workers. backend.ram_gb 278 GiB request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.gpu_service_lifecycle target stopped; per-request guards — The valid pool excludes a request that crossed an external target start. system.page_cache dropped-before-load; one untimed warm request — sudo sync and vm.drop_caches=3 preceded model load. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. Host expert slabs were interleaved. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts and selected dense tensors. Results and evidence `glm52-colibri-scalar-omp40-exclusive-confirm-20260814` Task text generation Status VALID Measured 2026-08-14T23:14:21Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.2 tok/s Elapsed 186.963 s Evidence colibri/results/20260814T231500Z-glm52-v100-matched-scalar-t40-exclusive-confirm/README.md Benchmark notes. Pinned scalar package. The valid control pools four guarded samples from one invocation and one pre-restart sample from an independent matched invocation. A sample crossed a target restart and was rejected. Result notes. Valid decode samples were 0.172819770582, 0.170378613225, 0.171156859913, 0.170629355714, and 0.171199897280 tokens/s. Four passed target and consumer guards before and after each request; one completed before a separately recorded service restart. A 0.168748780526 tokens/s request crossed a target start and was rejected.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.2	NA	NA
`cvs40` Colibri GLM-5.2 targeted SSE4.1 W4A32 with two V100s and 40 threads.1 result · 8 techniques · 16 parameters Configuration ID glm52-colibri-2v100-sse41-omp40-coldwarm32 Created 2026-08-14T21:53:12Z Notes Matched to cbx40 except for the Colibri Nix closure and grouped W4A32 CPU function target. Applied optimizations Architecture-specific build The engine kept ARCH=x86-64; grouped W4A32 functions used a GCC SSE4.1 target and CUDA used sm_70. Asynchronous expert I/O PIPE=1 used eight workers for missing-expert service. GPU layer offload CUDA_DENSE=1 placed selected dense tensors on the two V100s. Kernel-path tuning The fused grouped-int4 CPU path unpacked nibbles with SSE4.1 and shared activation loads between gate and up. MoE expert placement Both V100s held a shared 58 GB usage-ranked expert tier. NUMA memory policy COLI_NUMA=1 interleaved host expert slabs across both sockets. Release GPU capacity Resident GPU services were released before the retained region. Weight quantization The target artifact used grouped-int4 expert weights with f32 scales. Uncommon parameters backend.cache_route disabled backend.cuda_dense enabled — CUDA_DENSE=1. backend.cuda_devices 0,1 devices — COLI_GPUS=0,1. backend.cuda_expert_gb 58 GB — Total budget across both V100s. backend.cuda_pipeline enabled — COLI_CUDA_PIPE=1. backend.cuda_release_host enabled backend.direct_io disabled — DIRECT=0. backend.draft disabled — DRAFT=0 and MTP=0. backend.grouped_w4a32_kernel sse4.1-function-target backend.omp_threads 40 threads — OMP_DYNAMIC=FALSE. backend.pipe enabled — PIPE=1 with eight workers. backend.ram_gb 278 GiB request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.page_cache dropped-before-load; one untimed warm request — sudo sync and vm.drop_caches=3 preceded model load. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. Host expert slabs were interleaved. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts and selected dense tensors. Results and evidence `glm52-colibri-sse41-omp40-confirm-20260814` Task text generation Status VALID Measured 2026-08-14T21:53:12Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.2 tok/s Elapsed 134.760 s Baseline cbx40 (+38.7%) Output matches baseline yes Evidence colibri/results/20260814T220000Z-glm52-v100-targeted-sse41-final-t40-confirm/README.md Benchmark notes. Matched targeted-function candidate. Temperature zero made decoding greedy. Exact text and canonical token agreement is checked both within the run and against the scalar control. Result notes. Decode samples were 0.237026502526, 0.237492671125, 0.237459186702, 0.239050372396, and 0.235181714622 tokens/s. Candidate and scalar text SHA-256 was 481c7b79d21c977626bd72488b5189bf6818ade1782a9b36280b48b5f0913bb6. Their canonical token SHA-256 was e6b16a313a2701b581aa4efbf35b2fcc9bd30ed80a8ffe3f174c1f6275a52111.
	Colibri	CPU + V100	4096	17	—	—	—	—	—	0.2	NA	NA
`colq1` Colibri GLM-5.2 quality baseline with 278 GiB RAM and a 29 GB V100 expert tier.1 result · 5 techniques · 20 parameters Configuration ID glm52-colibri-v100-quality-r278-v29-c4k Created 2026-08-05T15:49:01Z Notes One serial client, one KV slot, greedy decode, fixed upstream expert history, one warm request, and five retained samples. Applied optimizations Architecture-specific build CUDA was built for sm_70. ARCH=x86-64 supplied a compatible scalar CPU path for the Ivy Bridge host. MoE expert placement Usage-ranked placement kept 1,365 experts in VRAM and 402 in pinned host RAM. The warm host LRU held 10,497 more experts. NUMA memory policy COLI_NUMA=1 interleaved expert slabs across both R720 NUMA nodes. Release GPU capacity The normal model router and ComfyUI stopped for the run. The exit cleanup restored both services and the prior router model. Weight quantization The target artifact used grouped int4 weights. Its unused MTP head used int8 weights. Uncommon parameters backend.cache_route disabled backend.cuda_expert_gb 29 GB backend.cuda_pipeline enabled — COLI_CUDA_PIPE=1. backend.direct_io disabled backend.disk_pipeline disabled — The runtime profile reported PIPE=0. backend.draft disabled — DRAFT=0 and MTP=0. backend.expert_top_p 0 — No expert routing truncation. backend.kv_slots 1 slots backend.max_queue 1 requests backend.persist_kv disabled — KVSAVE=0. backend.policy quality backend.ram_gb 278 GiB node:r720.gpu_thermal_control inactive — This matches the retained state before, during, and after the run. request.concurrency 1 requests — One serial HTTP connection issued all samples. request.prompt Explain why matched benchmarks are necessary for performance work. — The raw completion prompt includes the GLM prefix and assistant template shown in request.json. request.top_p 1 system.gpu_services stopped — ComfyUI and the normal model router released GPU capacity. system.page_cache warm-not-dropped — One untimed warm request preceded the five retained samples. target.model_revision baae76c03d5fde34c7dca801371f8f6aa38e4412 target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; Colibri placed individual experts instead of complete layers. It used 1,365 experts in VRAM, 402 pinned experts in host RAM, and a dynamic host-RAM LRU. GPU 1 was not used. Results and evidence `glm52-colibri-v100-quality-20260805` Task text generation Status VALID Measured 2026-08-05T15:49:01Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 128 Decode 0.2 tok/s Elapsed 832.041 s Peak host memory 254.5 GiB Peak VRAM 28.3 GiB Evidence colibri/results/20260805T154901Z-glm52-v100-quality/README.md Benchmark notes. Colibri does not accept a per-request seed. Temperature zero made the request greedy. The five exact text and canonical token sequences matched. The backend profile starts after prefix processing and reports 128 completion tokens over 127 decode forwards. Result notes. The decode samples were 0.150228923058, 0.152305524883, 0.153838572859, 0.154508385701, and 0.156106698929 tokens/s. Elapsed time is the median Colibri profile window. Host memory is the transient unit MemoryPeak. VRAM is the maximum of two retained nvidia-smi snapshots, not a continuous peak monitor. There was no separate matched baseline output; exact agreement applies across the five samples.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.7	NA	NA
`r4fs1` PR 1286 scalar full model, async I/O and non-streaming client.1 result · 2 techniques · 2 parameters Configuration ID glm52-colibri-pr1286-scalar-pipe1-ns32 Created 2026-08-30T21:58:16Z Notes Provisional because the client did not retain TTFT. Applied optimizations Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.cuda_dense enabled backend.pipe enabled — Eight workers. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts. Results and evidence `pr1286-full-s-pipe1-ns` Task text generation Status PROVISIONAL Measured 2026-08-30T21:58:16Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.7 tok/s Elapsed 43.986 s Output matches baseline yes Evidence colibri/results/20260830T215816Z-glm52-v100-pr1286-scalar-t40-confirm/README.md Benchmark notes. Valid throughput and exactness, but no retained TTFT. Result notes. Provisional: no retained TTFT. Range 0.720266-0.728880 tok/s.
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.4	NA	NA
`r4fs0` PR 1286 scalar full model with matched streaming timing.1 result · 2 techniques · 6 parameters Configuration ID glm52-colibri-pr1286-scalar-stream32 Created 2026-08-30T23:16:05Z Notes PIPE=0 and CUDA_DENSE=0; authoritative matched scalar control. Applied optimizations Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.cuda_dense disabled backend.pipe disabled request.client_protocol OpenAI completions SSE — Unbuffered curl event stream. system.page_cache_state dropped before model load system.prefix_cache_state full hit: 17/17 tokens; prefill 0 tokens — One KV slot and one untimed warm request. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Restored before and after the run. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts. Results and evidence `pr1286-full-s-stream-final` Task text generation Status VALID Measured 2026-08-30T23:16:05Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.4 tok/s Time to first token 2005.4 ms Elapsed 76.751 s Output matches baseline yes Evidence colibri/results/20260830T231605Z-glm52-v100-pr1286-scalar-t40-stream-final/README.md Benchmark notes. Authoritative matched streaming scalar control. Five exact responses. Result notes. Matched scalar range was 0.409296-0.418602 tok/s.
	Colibri	CPU + V100(2)	4096	17	—	?	—	—	—	—	NA	NA
	Colibri	CPU + V100(2)	4096	17	—	?	—	—	—	—	NA	NA
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	0.7	NA	NA
`r4fv0` PR 1286 rows4 full model with matched streaming timing.3 results · 4 techniques · 6 parameters Configuration ID glm52-colibri-pr1286-rows4-stream32 Created 2026-08-30T22:21:58Z Notes Used by two invalid client attempts and the final valid candidate. Applied optimizations Architecture-specific build SSE4.1 is enabled while AVX2 and FMA remain disabled for Ivy Bridge. Kernel-path tuning Four output rows are vectorized without changing a row reduction. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.cuda_dense disabled backend.pipe disabled request.client_protocol OpenAI completions SSE — Unbuffered curl event stream. system.page_cache_state dropped before model load system.prefix_cache_state full hit: 17/17 tokens; prefill 0 tokens — One KV slot and one untimed warm request. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Restored before and after each attempt. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts. Results and evidence `pr1286-full-v-stream-final` Task text generation Status VALID Measured 2026-08-30T22:56:36Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.7 tok/s Time to first token 1170.3 ms Elapsed 42.712 s Baseline r4fs0 (+79.7%) Output matches baseline yes Evidence colibri/results/20260830T225636Z-glm52-v100-pr1286-sse41-rows4-t40-stream-final/README.md Benchmark notes. Authoritative matched streaming candidate. Five exact responses. Result notes. Candidate sample range was wide; report the complete range. `pr1286-full-v-status-invalid` Task text generation Status INVALID Measured 2026-08-30T22:40:34Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Evidence colibri/results/20260830T224034Z-glm52-v100-pr1286-sse41-rows4-t40-stream-confirm/README.md Benchmark notes. SSE completed, but the client did not consume curl status metadata. Result notes. Rejected even though the sample reached the SSE done event. `pr1286-full-v-connect-invalid` Task text generation Status INVALID Measured 2026-08-30T22:21:58Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Evidence colibri/results/20260830T222158Z-glm52-v100-pr1286-sse41-rows4-t40-stream/README.md Benchmark notes. Python direct connection timed out before sample 1. Result notes. Rejected; curl health, smoke, and warm-up had succeeded.
	Colibri	CPU + V100(2)	4096	17	—	?	—	—	—	—	NA	NA
	Colibri	CPU + V100(2)	4096	17	—	—	—	—	—	1.2	NA	NA
`r4fv1` PR 1286 rows4 full model, async I/O and non-streaming client.2 results · 4 techniques · 2 parameters Configuration ID glm52-colibri-pr1286-rows4-pipe1-ns32 Created 2026-08-30T21:23:56Z Notes Used by one invalid parser attempt and one provisional valid run. Applied optimizations Architecture-specific build SSE4.1 is enabled while AVX2 and FMA remain disabled for Ivy Bridge. Kernel-path tuning Four output rows are vectorized without changing a row reduction. Release GPU capacity The benchmark stopped resident GPU consumers and restored them. Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.cuda_dense enabled backend.pipe enabled — Eight workers. Node placement r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 0 held usage-ranked experts. r720 — target; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; GPU 1 held usage-ranked experts. Results and evidence `pr1286-full-v-pipe1-ns` Task text generation Status PROVISIONAL Measured 2026-08-30T21:41:02Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 1.2 tok/s Elapsed 27.289 s Baseline r4fs1 (+61.2%) Output matches baseline yes Evidence colibri/results/20260830T214102Z-glm52-v100-pr1286-sse41-rows4-t40-confirm/README.md Benchmark notes. Valid throughput and exactness, but no retained TTFT. Result notes. Provisional: no retained TTFT. Range 1.016938-1.185580 tok/s. `pr1286-full-v-pipe1-invalid` Task text generation Status INVALID Measured 2026-08-30T21:23:56Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Evidence colibri/results/20260830T212356Z-glm52-v100-pr1286-sse41-rows4-t40/README.md Benchmark notes. Profile authentication parser failure. Result notes. Rejected before a complete retained set.
	Colibri	CPU + V100(2)	4096	17	—	Local GPU MTP	-33.5%	—	—	0.4	NA	NA
`col2m` Colibri GLM-5.2 dual-V100 MTP width-one screen.1 result · 5 techniques · 13 parameters Configuration ID glm52-colibri-2v100-r278-v58-pipe-dense-mtp1 Created 2026-08-06T07:37:53Z Notes Matched requested settings to col2g except for MTP=1 and DRAFT=1. The MTP head reduced the realized hot tier by 34 experts. Applied optimizations Asynchronous expert I/O PIPE=1 remained enabled. GPU layer offload CUDA_DENSE=1 placed dense tensors across both V100s. MoE expert placement Both V100s held 2,434 usage-ranked experts in 52.13 GB total. Speculative decoding The native int8 MTP head proposed one token per verification step. Speculative policy tuning Width one was measured on both V100s and rejected. Uncommon parameters backend.cache_route disabled backend.cuda_dense enabled — CUDA_DENSE=1. backend.cuda_devices 0,1 devices — COLI_GPUS=0,1. backend.cuda_expert_gb 58 GB — Total budget across both V100s. backend.cuda_release_host enabled — GPU-resident expert host copies were released. backend.direct_io disabled backend.draft mtp-width-1 tokens — MTP=1 and DRAFT=1. backend.pipe enabled — PIPE=1 with eight workers. backend.ram_gb 278 GiB request.concurrency 1 requests request.prompt Explain why matched benchmarks are necessary for performance work. system.page_cache warm-not-dropped — One untimed warm request preceded the retained screen. target.usage_history_sha256 5772a27e700b21909cf78461c45499d4f3e955fce796232920dd694357a77855 — Pinned upstream history restored before the run. Node placement r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 20 threads; NUMA interleave; GPU 0 held 1,418 usage-ranked experts in 30.47 GB. r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 20 threads; NUMA interleave; GPU 1 held 1,016 usage-ranked experts in 21.66 GB plus selected dense and native MTP tensors. Results and evidence `glm52-colibri-2v100-mtp1-screen-20260806` Task text generation Status PROVISIONAL Measured 2026-08-06T07:37:53Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.4 tok/s Elapsed 87.429 s Baseline col2g (-51.3%) Output matches baseline yes Evidence colibri/results/20260806T073753Z-glm52-v100-dual-mtp1-screen/README.md Benchmark notes. Rejected short screen. Width-one MTP kept exact output but reduced throughput by 51.32% relative to the dual-V100 target-only screen. Result notes. MTP emitted 32 tokens in 22 target forwards. It fetched 4.900 GB of experts and incurred 16.074 seconds of felt read wait.
	Colibri	CPU + V100	4096	17	—	Local GPU MTP	-77.8%	—	—	0.1	NA	NA
`colm3` Colibri MTP width three on the direct-I/O CUDA-dense profile.1 result · 2 techniques · 4 parameters Configuration ID glm52-colibri-v100-r278-v29-pipe-direct-dense-mtp3 Created 2026-08-06T00:38:24Z Notes Cache routing was disabled; the native int8 MTP head proposed up to three tokens. Applied optimizations Speculative decoding The native int8 MTP head proposed up to three tokens. Speculative policy tuning Width three was measured and rejected. Uncommon parameters backend.cuda_dense enabled backend.direct_io enabled backend.draft mtp-width-3 tokens — MTP=1 and DRAFT=3. backend.pipe enabled Node placement r720 — target_and_draft; CPU+CUDA; Tesla PG500-216 (V100 32GB); 78 target layers; 40 threads; NUMA interleave; The native MTP head and target shared the local host and one V100. Results and evidence `glm52-colibri-mtp3-screen-20260806` Task text generation Status PROVISIONAL Measured 2026-08-06T00:38:24Z Statistic single; 1 sample Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Context window 4096 Tested depth 0 Prompt tokens 17 Generated tokens 32 Decode 0.1 tok/s Elapsed 231.801 s Baseline colpdd Output matches baseline yes Evidence colibri/results/20260806T003824Z-glm52-v100-screen-mtp3 Benchmark notes. Rejected short screen. MTP kept exact output but widened expert work enough to reduce throughput. Result notes. The run emitted 32 tokens in 21 target forwards but performed about 1,574.5 expert loads per emitted token.

Transport and other workloads

Select any result row for complete configuration details. Non-model workloads use task-specific metrics. Collective rows show median out-of-place values; tensor proof rows show one eight-block step; host-device transfer rows show the median rate of an 8 GiB copy in each direction.

Cfg	Workload	Backend	Hosts	Accelerators	Interconnect	Validity	Samples	4 KiB us	16 MiB GB/s	Step ms	Steps/s	8 GiB H2D GB/s	8 GiB D2H GB/s	Source
	Eight-block tensor-parallel MLP	CUDA/NCCL	r720	Tesla PG500-216 (V100 32GB)	Local V100 HBM	valid	5	—	—	2.471	404.771	—	—	distributed/results/20260811T055619Z-v100-p40-tp-control
`tpcv` Both tensor shards execute locally on one V100.1 result · 0 techniques · 5 parameters Configuration ID tpmlp-v100-control-h4096-i8192-l8 Created 2026-08-11T05:56:27Z Notes Deterministic output reference with no inter-rank collective. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.virtual_shards 2 shards — Both virtual shards executed on one V100. target.output_hash 1d00c08067be8383 — FNV-1a over the raw FP16 output. workload.block_count 8 blocks — Blocks per measured step. workload.hidden_width 4096 elements — FP16 replicated activation width. workload.intermediate_width 8192 elements — Two equal virtual shards. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; physical GPU 1; One process executed both 64 MiB virtual weight shards. Results and evidence `tp-mlp-v100-control-20260811` Task other Status VALID Measured 2026-08-11T05:56:27Z Statistic median; 5 samples Model Eight-block tensor-parallel MLP Backend CUDA/NCCL Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Evidence distributed/results/20260811T055619Z-v100-p40-tp-control Benchmark notes. One physical V100 executed both virtual shards. Each process used three warm steps and ten measured steps. Result notes. Step latency samples: 2.466645, 2.493543, 2.471930, 2.470532, and 2.452803 ms.
	Eight-block tensor-parallel MLP	CUDA/NCCL	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; host staging	valid	5	—	—	3.344	299.072	—	—	distributed/results/20260811T055619Z-v100-p40-tp-roce-host
`tprh` V100/P40 tensor parallelism over host-staged RoCE.1 result · 1 technique · 6 parameters Configuration ID tpmlp-v100-p40-roce-host-h4096-i8192-l8 Created 2026-08-11T05:56:46Z Notes Two physical ranks; NCCL NET/IB without GDRDMA. Applied optimizations Tensor parallelism Two physical weight shards executed in parallel. Each of eight row-parallel projections ended with a host-staged RoCE all-reduce. Uncommon parameters network.gpudirect disabled — NCCL_NET_GDR_LEVEL=LOC. network.transport NCCL NET/IB — RoCE v1, mlx4_0 port 2, GID index 0, MTU 9000. target.output_hash 1d00c08067be8383 — Matched the local reference. workload.block_count 8 blocks — Eight all-reduces per step. workload.hidden_width 4096 elements — The all-reduce payload was 8 KiB per block. workload.intermediate_width 8192 elements — Split 1:1. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Rank 0; one 64 MiB physical weight shard. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Rank 1; one 64 MiB physical weight shard. Results and evidence `tp-mlp-v100-p40-roce-host-20260811` Task other Status VALID Measured 2026-08-11T05:56:46Z Statistic median; 5 samples Model Eight-block tensor-parallel MLP Backend CUDA/NCCL Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline tpcv (-26.1%) Output matches baseline yes Evidence distributed/results/20260811T055619Z-v100-p40-tp-roce-host Benchmark notes. Every raw trace used NET/IB and no trace contained GDRDMA. Both ranks and the local reference produced the same output hash. Result notes. Step latency samples: 3.344095, 3.343678, 3.342053, 4.336416, and 3.340907 ms.
	Eight-block tensor-parallel MLP	CUDA/NCCL	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; GPUDirect RDMA	valid	5	—	—	3.342	299.244	—	—	distributed/results/20260811T055619Z-v100-p40-tp-roce-gdr
`tprg` V100/P40 tensor parallelism over RoCE GPUDirect RDMA.1 result · 1 technique · 7 parameters Configuration ID tpmlp-v100-p40-roce-gdr-h4096-i8192-l8 Created 2026-08-11T05:57:15Z Notes Two physical ranks; every NCCL send and receive used GDRDMA. Applied optimizations Tensor parallelism Two physical weight shards executed in parallel. All eight all-reduces used GDRDMA for sends and receives. Uncommon parameters network.gpudirect send_and_receive — NCCL_NET_GDR_LEVEL=PHB and NCCL_NET_GDR_READ=1. network.transport NCCL NET/IB/GDRDMA — RoCE v1, mlx4_0 port 2, GID index 0, MTU 9000. system.rank_count 2 ranks — One MPI/NCCL rank per host. target.output_hash 1d00c08067be8383 — Matched the local reference. workload.block_count 8 blocks — Eight all-reduces per step. workload.hidden_width 4096 elements — The all-reduce payload was 8 KiB per block. workload.intermediate_width 8192 elements — Split 1:1. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Rank 0; one 64 MiB physical weight shard. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Rank 1; one 64 MiB physical weight shard. Results and evidence `tp-mlp-v100-p40-roce-gdr-20260811` Task other Status VALID Measured 2026-08-11T05:57:15Z Statistic median; 5 samples Model Eight-block tensor-parallel MLP Backend CUDA/NCCL Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline tpcv (-26.1%) Output matches baseline yes Evidence distributed/results/20260811T055619Z-v100-p40-tp-roce-gdr Benchmark notes. Every raw trace identified mlx4_0:2/RoCE. Every NCCL send and receive channel used NET/IB/0/GDRDMA. Result notes. Step latency samples: 3.343703, 3.341360, 3.341751, 3.346110, and 3.333532 ms.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260814T212000Z-final-sse41-grouped-int4-t40/README.md
`g4s40` Colibri grouped W4A32 scalar CPU kernel with 40 OpenMP threads.1 result · 1 technique · 6 parameters Configuration ID colibri-g4-cpu-scalar-t40 Created 2026-08-14T21:13:32Z Notes Generic x86-64 control, fixed production shape, and interleaved NUMA allocation. Applied optimizations Weight quantization The fixed tensor uses packed grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_w4a32_kernel scalar backend.omp_dynamic disabled request.input_seed 0x902a371d — Fixed xorshift32 input. request.iterations_per_sample 100 calls request.tensor_shape S1-I6144-O2048-g64-gate-plus-up system.service_state router and comfyui stopped; restored after run Node placement r720 — target; CPU; 40 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `colibri-g4-scalar-t40-20260814` Task other Status VALID Measured 2026-08-14T21:13:32Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Evidence colibri/results/20260814T212000Z-final-sse41-grouped-int4-t40/README.md Benchmark notes. Synthetic fixed-seed tensors use the model production shape S=1, I=6144, O=2048, group size 64, fused gate plus up. Each sample contains 100 calls. Result notes. Scalar median latency was 6.274146 milliseconds per fused call.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260814T212000Z-final-sse41-grouped-int4-t40/README.md
`g4v40` Colibri grouped W4A32 SSE4.1 CPU kernel with 40 OpenMP threads.1 result · 3 techniques · 6 parameters Configuration ID colibri-g4-cpu-sse41-t40 Created 2026-08-14T21:13:32Z Notes Matched to g4s40 except for the grouped-kernel function target. Applied optimizations Architecture-specific build A GCC SSE4.1 function target specializes only grouped W4A32. Kernel-path tuning Packed nibbles are unpacked with SSE4.1 and gate plus up share each activation load. Weight quantization The fixed tensor uses packed grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_w4a32_kernel sse4.1-function-target backend.omp_dynamic disabled request.input_seed 0x902a371d — Fixed xorshift32 input. request.iterations_per_sample 100 calls request.tensor_shape S1-I6144-O2048-g64-gate-plus-up system.service_state router and comfyui stopped; restored after run Node placement r720 — target; CPU; 40 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `colibri-g4-sse41-t40-20260814` Task other Status VALID Measured 2026-08-14T21:13:32Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Baseline g4s40 (+34.2%) Evidence colibri/results/20260814T212000Z-final-sse41-grouped-int4-t40/README.md Benchmark notes. The harness alternated scalar and candidate order. Both executable contracts passed before timing. Result notes. SSE4.1 median latency was 4.676419 milliseconds per fused call. The scalar and SIMD accumulation orders differ, so the executable contract uses a magnitude-scaled numeric tolerance.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260814T212100Z-final-sse41-grouped-int4-t20/README.md
`g4s20` Colibri grouped W4A32 scalar CPU kernel with 20 OpenMP threads.1 result · 1 technique · 6 parameters Configuration ID colibri-g4-cpu-scalar-t20 Created 2026-08-14T21:13:49Z Notes One thread per physical R720 core with interleaved NUMA allocation. Applied optimizations Weight quantization The fixed tensor uses packed grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_w4a32_kernel scalar backend.omp_dynamic disabled request.input_seed 0x902a371d — Fixed xorshift32 input. request.iterations_per_sample 100 calls request.tensor_shape S1-I6144-O2048-g64-gate-plus-up system.service_state router and comfyui stopped; restored after run Node placement r720 — target; CPU; 20 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `colibri-g4-scalar-t20-20260814` Task other Status VALID Measured 2026-08-14T21:13:49Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Evidence colibri/results/20260814T212100Z-final-sse41-grouped-int4-t20/README.md Benchmark notes. Synthetic fixed-seed tensors use the same production shape. Each sample contains 100 calls. Result notes. Scalar median latency was 2.835388 milliseconds per fused call.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260814T212100Z-final-sse41-grouped-int4-t20/README.md
`g4v20` Colibri grouped W4A32 SSE4.1 CPU kernel with 20 OpenMP threads.1 result · 3 techniques · 6 parameters Configuration ID colibri-g4-cpu-sse41-t20 Created 2026-08-14T21:13:49Z Notes Matched to g4s20 except for the grouped-kernel function target. Applied optimizations Architecture-specific build A GCC SSE4.1 function target specializes only grouped W4A32. Kernel-path tuning Packed nibbles are unpacked with SSE4.1 and gate plus up share each activation load. Weight quantization The fixed tensor uses packed grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_w4a32_kernel sse4.1-function-target backend.omp_dynamic disabled request.input_seed 0x902a371d — Fixed xorshift32 input. request.iterations_per_sample 100 calls request.tensor_shape S1-I6144-O2048-g64-gate-plus-up system.service_state router and comfyui stopped; restored after run Node placement r720 — target; CPU; 20 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `colibri-g4-sse41-t20-20260814` Task other Status VALID Measured 2026-08-14T21:13:49Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Baseline g4s20 (+289.5%) Evidence colibri/results/20260814T212100Z-final-sse41-grouped-int4-t20/README.md Benchmark notes. The harness alternated scalar and candidate order. Both executable contracts passed before timing. Result notes. SSE4.1 median latency was 0.727938 milliseconds per fused call. This isolated result does not select the complete-model thread count.
	GLM-5.2	Colibri	r720	Tesla PG500-216 (V100 32GB)	—	valid	5	—	—	—	—	—	—	colibri/results/20260815T002617Z-g64-warp-v100/README.md
`g4vg` Colibri generic grouped-int4 CUDA kernel on one V100.1 result · 2 techniques · 4 parameters Configuration ID colibri-g4-v100-generic Created 2026-08-15T00:26:17Z Notes Fixed production shape with paired alternating measurements. Applied optimizations Architecture-specific build CUDA code was compiled for Volta sm_70. Weight quantization The fixed tensors use packed grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_g64_cuda_kernel generic request.iterations_per_sample 20 calls request.tensor_shape S1-D6144-I2048-g64-hidden-plus-down system.gpu_service_lifecycle target stopped; pair guards; restored after run Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave; Physical CUDA0 ran with all resident GPU services stopped. Results and evidence `colibri-g4-v100-generic-20260815` Task other Status VALID Measured 2026-08-15T00:26:17Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Evidence colibri/results/20260815T002617Z-g64-warp-v100/README.md Benchmark notes. Each sample contains 20 hidden-plus-down calls at S1-D6144-I2048-g64. Invocation order alternated by sample. Result notes. Generic median latency was 0.157491192 milliseconds per complete hidden-plus-down call.
	GLM-5.2	Colibri	r720	Tesla PG500-216 (V100 32GB)	—	valid	5	—	—	—	—	—	—	colibri/results/20260815T002617Z-g64-warp-v100/README.md
`g4vw` Colibri warp-per-G64 grouped-int4 CUDA kernel on one V100.1 result · 3 techniques · 5 parameters Configuration ID colibri-g4-v100-warp64 Created 2026-08-15T00:26:17Z Notes Matched to g4vg except for the kernel launch. Applied optimizations Architecture-specific build CUDA code was compiled for Volta sm_70. Kernel-path tuning One logical warp reduces one G64 dot product and applies its scale once; gate and up share activation loads. Weight quantization The fixed tensors use packed grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_g64_cuda_kernel warp-per-group backend.numeric_contract four shapes passed cases — Includes two tail shapes and deterministic repeat checking. request.iterations_per_sample 20 calls request.tensor_shape S1-D6144-I2048-g64-hidden-plus-down system.gpu_service_lifecycle target stopped; pair guards; restored after run Node placement r720 — target; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave; Physical CUDA0 ran with all resident GPU services stopped. Results and evidence `colibri-g4-v100-warp64-20260815` Task other Status VALID Measured 2026-08-15T00:26:17Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Baseline g4vg (+16.9%) Output matches baseline yes Evidence colibri/results/20260815T002617Z-g64-warp-v100/README.md Benchmark notes. The executable numeric contract passed before timing. Result notes. Warp-per-G64 median latency was 0.134707198 milliseconds. The CPU numeric reference passed; floating-point reduction order changes the bitwise output hash.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260830T212230Z-ivybridge-grouped-int4/README.md
`r4s20` PR 1286 scalar grouped-int4 kernel with 20 OpenMP threads.1 result · 1 technique · 2 parameters Configuration ID colibri-pr1286-g4-scalar-t20 Created 2026-08-30T21:22:32Z Notes One thread per physical Ivy Bridge core. Applied optimizations Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_int4_kernel scalar request.tensor_shape S1-I6144-O2048-g64-gate-plus-up Node placement r720 — target; CPU; 20 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `pr1286-g4-s20-20260830` Task other Status VALID Measured 2026-08-30T21:22:32Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Output matches baseline yes Evidence colibri/results/20260830T212230Z-ivybridge-grouped-int4/README.md Benchmark notes. Fixed S1-I6144-O2048-g64 gate-plus-up tensor; 100 calls per sample. Result notes. Median scalar latency 1.617267 ms; checksum b5253d3717f0ae12.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260830T212230Z-ivybridge-grouped-int4/README.md
`r4v20` PR 1286 exact rows4 SSE4.1 kernel with 20 OpenMP threads.1 result · 3 techniques · 2 parameters Configuration ID colibri-pr1286-g4-rows4-t20 Created 2026-08-30T21:22:32Z Notes Matched to r4s20 except for SSE4.1. Applied optimizations Architecture-specific build SSE4.1 is enabled while AVX2 and FMA remain disabled for Ivy Bridge. Kernel-path tuning Four output rows are vectorized without changing a row reduction. Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_int4_kernel sse4.1-rows4-exact request.tensor_shape S1-I6144-O2048-g64-gate-plus-up Node placement r720 — target; CPU; 20 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `pr1286-g4-v20-20260830` Task other Status VALID Measured 2026-08-30T21:22:32Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Baseline r4s20 (+115.4%) Output matches baseline yes Evidence colibri/results/20260830T212230Z-ivybridge-grouped-int4/README.md Benchmark notes. Matched rows4 candidate; all scalar and SIMD checksums match. Result notes. Median rows4 latency 0.750824 ms; bit-exact scalar checksum.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260830T212251Z-ivybridge-grouped-int4/README.md
`r4s40` PR 1286 scalar grouped-int4 kernel with 40 OpenMP threads.1 result · 1 technique · 2 parameters Configuration ID colibri-pr1286-g4-scalar-t40 Created 2026-08-30T21:22:53Z Notes All Ivy Bridge hardware threads. Applied optimizations Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_int4_kernel scalar request.tensor_shape S1-I6144-O2048-g64-gate-plus-up Node placement r720 — target; CPU; 40 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `pr1286-g4-s40-20260830` Task other Status VALID Measured 2026-08-30T21:22:53Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Output matches baseline yes Evidence colibri/results/20260830T212251Z-ivybridge-grouped-int4/README.md Benchmark notes. Fixed S1-I6144-O2048-g64 gate-plus-up tensor; 100 calls per sample. Result notes. Median scalar latency 1.858267 ms; checksum b5253d3717f0ae12.
	GLM-5.2	Colibri	r720	—	—	valid	5	—	—	—	—	—	—	colibri/results/20260830T212251Z-ivybridge-grouped-int4/README.md
`r4v40` PR 1286 exact rows4 SSE4.1 kernel with 40 OpenMP threads.1 result · 3 techniques · 2 parameters Configuration ID colibri-pr1286-g4-rows4-t40 Created 2026-08-30T21:22:53Z Notes Matched to r4s40 except for SSE4.1. Applied optimizations Architecture-specific build SSE4.1 is enabled while AVX2 and FMA remain disabled for Ivy Bridge. Kernel-path tuning Four output rows are vectorized without changing a row reduction. Weight quantization The artifact or fixed tensor uses grouped-int4 weights and f32 scales. Uncommon parameters backend.grouped_int4_kernel sse4.1-rows4-exact request.tensor_shape S1-I6144-O2048-g64-gate-plus-up Node placement r720 — target; CPU; 40 threads; NUMA interleave; OMP_DYNAMIC=FALSE; OMP_PROC_BIND=FALSE. Results and evidence `pr1286-g4-v40-20260830` Task other Status VALID Measured 2026-08-30T21:22:53Z Statistic median; 5 samples Model GLM-5.2 Backend Colibri Hosts r720 Hardware CPU Baseline r4s40 (+165.5%) Output matches baseline yes Evidence colibri/results/20260830T212251Z-ivybridge-grouped-int4/README.md Benchmark notes. Matched rows4 candidate; all scalar and SIMD checksums match. Result notes. Median rows4 latency 0.699801 ms; bit-exact scalar checksum.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:05:00.0 to 0000:42:00.0; SYS path without peer access	valid	5	17.90	10.65	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp0ip` CUDA0 to CUDA1 under interleave-all, through peer.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda0-to-cuda1-interleave-peer Created 2026-08-21T23:38:25Z Notes The transfer uses cudaMemcpyPeer, which lets the driver route and pipeline it. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Destination device. Results and evidence `gpu-peer-cuda0-cuda1-interleave-peer-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.024 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the peer path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 10.91, 10.94, 10.98, 11.27, and 11.30 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:05:00.0 to 0000:42:00.0; SYS path without peer access	valid	5	20.57	6.04	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp0is` CUDA0 to CUDA1 under interleave-all, through staged.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda0-to-cuda1-interleave-staged Created 2026-08-21T23:38:25Z Notes The transfer uses an explicit device-to-host then host-to-device copy through a pinned buffer, without pipelining. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Destination device. Results and evidence `gpu-peer-cuda0-cuda1-interleave-staged-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.043 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the staged path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 6.24, 6.29, 6.29, 6.30, and 6.31 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:05:00.0 to 0000:42:00.0; SYS path without peer access	valid	5	17.35	10.59	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp0np` CUDA0 to CUDA1 under NUMA node 0, through peer.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda0-to-cuda1-node0-peer Created 2026-08-21T23:38:25Z Notes The transfer uses cudaMemcpyPeer, which lets the driver route and pipeline it. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Destination device. Results and evidence `gpu-peer-cuda0-cuda1-node0-peer-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.024 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the peer path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 10.83, 10.93, 11.03, 11.29, and 11.29 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:05:00.0 to 0000:42:00.0; SYS path without peer access	valid	5	18.91	6.22	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp0ns` CUDA0 to CUDA1 under NUMA node 0, through staged.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda0-to-cuda1-node0-staged Created 2026-08-21T23:38:25Z Notes The transfer uses an explicit device-to-host then host-to-device copy through a pinned buffer, without pipelining. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Destination device. Results and evidence `gpu-peer-cuda0-cuda1-node0-staged-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.043 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the staged path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 6.26, 6.27, 6.27, 6.28, and 6.28 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:42:00.0 to 0000:05:00.0; SYS path without peer access	valid	5	15.52	10.51	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp1ip` CUDA1 to CUDA0 under interleave-all, through peer.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda1-to-cuda0-interleave-peer Created 2026-08-21T23:38:25Z Notes The transfer uses cudaMemcpyPeer, which lets the driver route and pipeline it. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Destination device. Results and evidence `gpu-peer-cuda1-cuda0-interleave-peer-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.024 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the peer path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 11.14, 11.15, 11.15, 11.26, and 11.34 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:42:00.0 to 0000:05:00.0; SYS path without peer access	valid	5	20.50	6.04	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp1is` CUDA1 to CUDA0 under interleave-all, through staged.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda1-to-cuda0-interleave-staged Created 2026-08-21T23:38:25Z Notes The transfer uses an explicit device-to-host then host-to-device copy through a pinned buffer, without pipelining. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Destination device. Results and evidence `gpu-peer-cuda1-cuda0-interleave-staged-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.043 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the staged path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 6.23, 6.23, 6.24, 6.24, and 6.25 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:42:00.0 to 0000:05:00.0; SYS path without peer access	valid	5	16.35	9.83	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp1np` CUDA1 to CUDA0 under NUMA node 0, through peer.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda1-to-cuda0-node0-peer Created 2026-08-21T23:38:25Z Notes The transfer uses cudaMemcpyPeer, which lets the driver route and pipeline it. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Destination device. Results and evidence `gpu-peer-cuda1-cuda0-node0-peer-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.026 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the peer path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 9.80, 10.40, 10.44, 10.48, and 10.62 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	GPU peer transfer	device-peer-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair, 0000:42:00.0 to 0000:05:00.0; SYS path without peer access	valid	5	18.75	6.10	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`gp1ns` CUDA1 to CUDA0 under NUMA node 0, through staged.1 result · 2 techniques · 0 parameters Configuration ID gpu-peer-cuda1-to-cuda0-node0-staged Created 2026-08-21T23:38:25Z Notes The transfer uses an explicit device-to-host then host-to-device copy through a pinned buffer, without pipelining. Applied optimizations GPU link topology Both cards now negotiate x16 on different root ports. GPU peer access The pair reports no peer access, so the driver stages the copy. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Source device. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Destination device. Results and evidence `gpu-peer-cuda1-cuda0-node0-staged-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model GPU peer transfer Backend device-peer-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.044 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. One directed copy through the staged path. Result notes. Seconds for one 256 MiB copy, median of five samples. Sorted rates: 5.95, 6.00, 6.11, 6.16, and 6.18 GB/s. Each arm copied a position-dependent pattern and compared the bytes at the destination.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	TCP over direct ConnectX-3 40 GbE	valid	5	78.14	1.71	—	—	—	—	distributed/results/20260809T081110Z-v100-p40-tcp-40gbe
`nctcp` Two-rank FP16 all-reduce over TCP on the direct 40 GbE link.1 result · 0 techniques · 10 parameters Configuration ID nccl-v100-p40-tcp-40gbe-fp16 Created 2026-08-09T08:11:21Z Notes Matched control for RoCE host-staging. NCCL RDMA was disabled. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.transport NET/Socket — NCCL_IB_DISABLE=1. network.interface roce0 — The name is shared by both direct-link ports. network.link_speed 40 Gbit/second network.mtu 9000 bytes system.rank_count 2 ranks — One rank on each host. workload.collective all_reduce workload.iterations 50 iterations workload.message_range 4096-16777216 bytes — Power-of-two sweep. workload.reduction sum workload.warmup_iterations 10 iterations Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1 at 0000:42:00.0; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0 at 0000:42:00.0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-tcp-40gbe-20260809` Task other Status VALID Measured 2026-08-09T08:11:21Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Evidence distributed/results/20260809T081110Z-v100-p40-tcp-40gbe Benchmark notes. Every retained trace used NET/Socket and no trace used NET/IB. The physical-link iperf3 control averaged 28.1 Gbit/s with zero retransmits. Metrics are medians of the five out-of-place rows. Result notes. The 4 KiB times were 117.60, 74.71, 72.64, 78.14, and 78.91 us. The 16 MiB bus bandwidth values were 1.71, 1.70, 1.75, 1.71, and 1.70 GB/s. Raw logs retain both out-of-place and in-place rows.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; host staging	valid	5	23.01	3.83	—	—	—	—	distributed/results/20260809T081110Z-v100-p40-roce-host
`ncroce` Two-rank FP16 all-reduce over RoCE v1 with host staging.1 result · 0 techniques · 16 parameters Configuration ID nccl-v100-p40-roce-v1-host-fp16 Created 2026-08-09T08:12:05Z Notes NCCL used mlx4_0 port 2 and GID index 1. GPUDirect RDMA was disabled. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.gdr disabled — NCCL_NET_GDR_LEVEL=LOC. backend.gin disabled; proxy type selected — NCCL_GIN_ENABLE=0 and NCCL_GIN_TYPE=2 avoided invalid P40 GDAKI auto-detection in NCCL 2.28.7. backend.transport NET/IB — No GDRDMA marker appeared. network.gid_index 1 — IPv4-mapped GIDs for 10.42.0.1 and 10.42.0.2. network.hca_port mlx4_0:2 network.interface roce0 — The name is shared by both direct-link ports. network.link_speed 40 Gbit/second network.mtu 9000 bytes — RDMA active MTU was 4096 bytes. network.roce_version 1 — Plain ConnectX-3; direct layer-2 link. system.memlock unlimited — The benchmark wrapper raised only the MPI rank process limit before NCCL registered verbs memory. system.rank_count 2 ranks — One rank on each host. workload.collective all_reduce workload.iterations 50 iterations workload.message_range 4096-16777216 bytes — Power-of-two sweep. workload.reduction sum workload.warmup_iterations 10 iterations Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1 at 0000:42:00.0; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0 at 0000:42:00.0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-roce-host-20260809` Task other Status VALID Measured 2026-08-09T08:12:05Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nctcp Evidence distributed/results/20260809T081110Z-v100-p40-roce-host Benchmark notes. Every retained trace used NET/IB on mlx4_0 port 2 and GID index 1, and no trace used GDRDMA. The 1 MiB verbs preflight measured 38.523 Gbit/s. Metrics are medians of five out-of-place rows. Result notes. The 4 KiB times were 23.13, 22.38, 23.13, 23.01, and 22.43 us. The 16 MiB bus bandwidth values were 3.80, 3.86, 3.83, 3.83, and 3.86 GB/s. Raw logs retain both out-of-place and in-place rows.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	PRRTE control launch toward direct RoCE link	invalid	6	—	—	—	—	—	—	distributed/results/20260810T211644Z-v100-p40-roce-gdr-launch-screens
`ncgdrlx` Rejected PRRTE launch configurations before the control-path fix.1 result · 0 techniques · 2 parameters Configuration ID nccl-v100-p40-gdr-launch-invalid Created 2026-08-10T21:16:44Z Notes The remote daemon bound its WireGuard address while connecting to the RoCE controller address. No NCCL measurement completed. Applied optimizations No applied technique is cataloged. Uncommon parameters network.controller_network 10.42.0.0/30 — Remote source was incorrectly 10.77.77.13. runtime.prrte 4.1.0-unpatched — Remote daemon MCA interface selection was not applied. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1 at 0000:42:00.0; intended rank did not launch. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0 at 0000:42:00.0; intended rank did not launch. Results and evidence `nccl-v100-p40-gdr-launch-invalid-20260810` Task other Status INVALID Measured 2026-08-10T21:16:44Z Statistic single; 6 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Evidence distributed/results/20260810T211644Z-v100-p40-roce-gdr-launch-screens Benchmark notes. Six rejected launch screens are retained. They cover a stale SSH host key and PRRTE 4.1.0 remote-daemon interface selection before the maintained-branch initialization fix. They produced no collective measurements and are excluded from comparisons. Result notes. The remote daemon selected 10.77.77.13 as its source while the controller advertised 10.42.0.1. The retained screens document each rejected configuration and the final successful probe.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; GPUDirect RDMA	valid	1	27.25	—	—	—	—	—	distributed/results/20260810T211644Z-v100-p40-roce-gdr-launch-screens/nccl-gdr-screen-20260810T211644Z-7
`ncgdrp` One-sample 4 KiB through 64 KiB GDR transport validation probe.1 result · 0 techniques · 6 parameters Configuration ID nccl-v100-p40-roce-v1-gdr-phb-probe Created 2026-08-10T22:16:25Z Notes Every NCCL channel used mlx4_0 port 2 with the GDRDMA marker. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.transport NET/IB/0/GDRDMA — NCCL_NET_GDR_LEVEL=PHB. network.gid_index 0 — RoCE v1 link-local GID on both hosts. workload.collective all_reduce workload.iterations 20 iterations workload.message_range 4096-65536 bytes — Power-of-two sweep. workload.warmup_iterations 5 iterations Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1 at 0000:42:00.0; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0 at 0000:42:00.0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-roce-gdr-probe-20260810` Task other Status VALID Measured 2026-08-10T22:16:25Z Statistic single; 1 sample Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Evidence distributed/results/20260810T211644Z-v100-p40-roce-gdr-launch-screens/nccl-gdr-screen-20260810T211644Z-7 Benchmark notes. A short 4 KiB through 64 KiB probe proved GDRDMA before the matched five-sample series. Result notes. One diagnostic sample; all channels reported GDRDMA. It is not used for the five-sample median comparison.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; host staging	valid	5	22.24	3.87	—	—	—	—	distributed/results/20260810T211644Z-v100-p40-roce-host-patched
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; host staging	valid	5	22.82	3.84	—	—	—	—	distributed/results/20260810T225836Z-v100-p40-roce-host-control
`nchost0` Two-rank FP16 all-reduce over RoCE v1 with host staging and GID 0.2 results · 0 techniques · 17 parameters Configuration ID nccl-v100-p40-roce-v1-host-gid0-fp16 Created 2026-08-10T22:16:46Z Notes Matched baseline for the PHB-level GPUDirect RDMA run. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.gdr disabled — NCCL_NET_GDR_LEVEL=LOC. backend.gin disabled; proxy type selected — NCCL_GIN_ENABLE=0 and NCCL_GIN_TYPE=2. backend.transport NET/IB — No GDRDMA marker appeared. network.gid_index 0 — RoCE v1 link-local GID on both hosts. network.hca_port mlx4_0:2 network.interface roce0 — 10.42.0.1/30 and 10.42.0.2/30. network.link_speed 40 Gbit/second network.mtu 9000 bytes — RDMA active MTU was 4096 bytes. network.roce_version 1 — Plain ConnectX-3; direct layer-2 link. runtime.prrte_if_include 10.42.0.0/30 — PRRTE process control also used the dedicated link. system.memlock unlimited — Raised for each MPI rank. system.rank_count 2 ranks — One rank on each host. workload.collective all_reduce workload.iterations 50 iterations workload.message_range 4096-16777216 bytes — Power-of-two sweep. workload.reduction sum workload.warmup_iterations 10 iterations Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1; one MPI/NCCL rank. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1 at 0000:42:00.0; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0 at 0000:42:00.0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-roce-host-direction-20260810` Task other Status VALID Measured 2026-08-10T23:00:54Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Evidence distributed/results/20260810T225836Z-v100-p40-roce-host-control Benchmark notes. All send and receive channels used NET/IB without GDRDMA. Result notes. The five 16 MiB bus bandwidth values were 3.83, 3.86, 3.82, 3.85, and 3.84 GB/s. `nccl-v100-p40-roce-host-gid0-20260810` Task other Status VALID Measured 2026-08-10T22:16:46Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Evidence distributed/results/20260810T211644Z-v100-p40-roce-host-patched Benchmark notes. Every retained trace used NET/IB on mlx4_0 port 2 and GID index 0, and no trace used GDRDMA. Metrics are medians of five out-of-place rows. Result notes. The 4 KiB times were 22.90, 21.49, 22.15, 22.83, and 22.24 us. The 16 MiB bus bandwidth values were 3.85, 3.87, 3.83, 3.88, and 3.87 GB/s.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 over direct ConnectX-3 40 GbE; PHB GPUDirect RDMA	valid	5	21.75	2.31	—	—	—	—	distributed/results/20260810T211644Z-v100-p40-roce-gdr-phb
`ncgdr0` Two-rank FP16 all-reduce over RoCE v1 with PHB-level GDR and GID 0.1 result · 1 technique · 17 parameters Configuration ID nccl-v100-p40-roce-v1-gdr-phb-gid0-fp16 Created 2026-08-10T22:17:17Z Notes All channels used NET/IB/0/GDRDMA, but large-message performance regressed against the matched host-staged baseline. Applied optimizations GPUDirect RDMA GDRDMA was active on every channel. It reduced median 4 KiB latency by 2.2%, but reduced median 16 MiB bus bandwidth by 40.3% on this V100/P40 and ConnectX-3 path. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL=PHB. backend.gin disabled; proxy type selected — NCCL_GIN_ENABLE=0 and NCCL_GIN_TYPE=2. backend.transport NET/IB/0/GDRDMA — All channels carried the marker. network.gid_index 0 — RoCE v1 link-local GID on both hosts. network.hca_port mlx4_0:2 network.interface roce0 — 10.42.0.1/30 and 10.42.0.2/30. network.link_speed 40 Gbit/second network.mtu 9000 bytes — RDMA active MTU was 4096 bytes. network.roce_version 1 — Plain ConnectX-3; direct layer-2 link. runtime.prrte_if_include 10.42.0.0/30 — PRRTE process control also used the dedicated link. system.memlock unlimited — Raised for each MPI rank. system.rank_count 2 ranks — One rank on each host. workload.collective all_reduce workload.iterations 50 iterations workload.message_range 4096-16777216 bytes — Power-of-two sweep. workload.reduction sum workload.warmup_iterations 10 iterations Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1 at 0000:42:00.0; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0 at 0000:42:00.0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-roce-gdr-phb-20260810` Task other Status VALID Measured 2026-08-10T22:17:17Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nchost0 Evidence distributed/results/20260810T211644Z-v100-p40-roce-gdr-phb Benchmark notes. Every retained trace used NET/IB/0/GDRDMA on both hosts. Metrics are medians of five out-of-place rows. Result notes. The 4 KiB times were 21.25, 34.10, 24.46, 20.43, and 21.75 us. The 16 MiB bus bandwidth values were 2.31, 2.31, 2.31, 2.31, and 2.32 GB/s.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1; direct GPU receives and host-staged sends	valid	5	22.25	4.31	—	—	—	—	distributed/results/20260810T225836Z-v100-p40-roce-gdr-write-only
`ncgdw1` Two-rank FP16 all-reduce with direct GDR receives and staged sends.1 result · 1 technique · 4 parameters Configuration ID nccl-v100-p40-roce-v1-gdr-write-gid0-fp16 Created 2026-08-10T23:01:25Z Notes NCCL_NET_GDR_LEVEL=PHB and NCCL_NET_GDR_READ=0. Applied optimizations GPUDirect RDMA Direct receives and staged sends raised median 16 MiB bus bandwidth from 3.84 to 4.31 GB/s. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL=PHB. backend.gdr_read disabled — NCCL_NET_GDR_READ=0; sends used host staging. workload.iterations 50 iterations — Ten warm-up iterations. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Direct GDR receive and staged send. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Direct GDR receive and staged send. Results and evidence `nccl-v100-p40-roce-gdr-write-20260810` Task other Status VALID Measured 2026-08-10T23:01:25Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nchost0 Evidence distributed/results/20260810T225836Z-v100-p40-roce-gdr-write-only Benchmark notes. Receive channels used GDRDMA. Send channels did not. NCCL_NET_GDR_READ=0 was explicit. Result notes. The five 16 MiB bus bandwidth values were 4.31, 4.31, 4.30, 4.30, and 4.31 GB/s.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1; direct GPU receives and sends	valid	5	21.81	2.31	—	—	—	—	distributed/results/20260810T225836Z-v100-p40-roce-gdr-read-write
`ncgdr1` Two-rank FP16 all-reduce with direct GDR receives and sends.1 result · 1 technique · 4 parameters Configuration ID nccl-v100-p40-roce-v1-gdr-rw-gid0-fp16 Created 2026-08-10T23:01:52Z Notes NCCL_NET_GDR_LEVEL=PHB and NCCL_NET_GDR_READ=1. Applied optimizations GPUDirect RDMA Direct receives and sends reduced median 16 MiB bus bandwidth to 2.31 GB/s. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL=PHB. backend.gdr_read enabled — NCCL_NET_GDR_READ=1. workload.iterations 50 iterations — Ten warm-up iterations. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Direct GDR receive and send. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Direct GDR receive and send. Results and evidence `nccl-v100-p40-roce-gdr-rw-20260810` Task other Status VALID Measured 2026-08-10T23:01:52Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nchost0 Evidence distributed/results/20260810T225836Z-v100-p40-roce-gdr-read-write Benchmark notes. All send and receive channels used GDRDMA. NCCL_NET_GDR_READ=1 was explicit. Result notes. The five 16 MiB bus bandwidth values were 2.31, 2.31, 2.32, 2.31, and 2.31 GB/s.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 port 2; host staging	valid	5	23.14	3.84	—	—	—	—	distributed/results/20260814T154150Z-host-rail0
`nhr0` Two-rank FP16 all-reduce with host staging on port 2.1 result · 0 techniques · 13 parameters Configuration ID nccl-v100-p40-dual-host-r0-fp16 Created 2026-08-14T15:42:06Z Notes Single-port matched control. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.gdr disabled — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read disabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:2 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-host-r0-20260814` Task other Status VALID Measured 2026-08-14T15:42:06Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Evidence distributed/results/20260814T154150Z-host-rail0 Benchmark notes. Correct output and the requested NET/IB transport in every sample. Result notes. Single-port host-staging control.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 port 1; host staging	valid	5	22.50	3.84	—	—	—	—	distributed/results/20260814T154150Z-host-rail1
`nhr1` Two-rank FP16 all-reduce with host staging on port 1.1 result · 0 techniques · 13 parameters Configuration ID nccl-v100-p40-dual-host-r1-fp16 Created 2026-08-14T15:42:29Z Notes Single-port matched control. Applied optimizations No applied technique is cataloged. Uncommon parameters backend.gdr disabled — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read disabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:1 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce1; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce1; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-host-r1-20260814` Task other Status VALID Measured 2026-08-14T15:42:29Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nhr0 Evidence distributed/results/20260814T154150Z-host-rail1 Benchmark notes. Correct output and the requested NET/IB transport in every sample. Result notes. Second single-port host-staging control.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 fused ports; host staging	valid	5	28.84	3.84	—	—	—	—	distributed/results/20260814T154150Z-host-dual
`nh2` Two-rank FP16 all-reduce with host staging on fused ports.1 result · 1 technique · 13 parameters Configuration ID nccl-v100-p40-dual-host-dual-fp16 Created 2026-08-14T15:42:52Z Notes Both HCA ports had nonzero counters. Applied optimizations Multi-rail transport Both ports carried data, but 16 MiB bandwidth did not improve. Uncommon parameters backend.gdr disabled — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read disabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:1,mlx4_0:2 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-host-dual-20260814` Task other Status VALID Measured 2026-08-14T15:42:52Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nhr0 Evidence distributed/results/20260814T154150Z-host-dual Benchmark notes. Both ports had positive data counters in every valid sample. Result notes. Dual rail did not improve 16 MiB host staging.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 port 2; direct receive and staged send	valid	5	21.84	4.31	—	—	—	—	distributed/results/20260814T154150Z-hybrid-rail0
`nwr0` Two-rank FP16 all-reduce with direct receives on port 2.1 result · 1 technique · 13 parameters Configuration ID nccl-v100-p40-dual-hybrid-r0-fp16 Created 2026-08-14T15:43:15Z Notes NCCL_NET_GDR_READ=0. Applied optimizations GPUDirect RDMA Direct receives used GDRDMA; sends used host staging. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read disabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:2 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-hybrid-r0-20260814` Task other Status VALID Measured 2026-08-14T15:43:15Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nhr0 Evidence distributed/results/20260814T154150Z-hybrid-rail0 Benchmark notes. Correct output; receive used GDRDMA and send used host staging. Result notes. Single-port direct-receive and staged-send control.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 port 1; direct receive and staged send	valid	5	22.12	4.26	—	—	—	—	distributed/results/20260814T154150Z-hybrid-rail1
`nwr1` Two-rank FP16 all-reduce with direct receives on port 1.1 result · 1 technique · 13 parameters Configuration ID nccl-v100-p40-dual-hybrid-r1-fp16 Created 2026-08-14T15:43:39Z Notes NCCL_NET_GDR_READ=0. Applied optimizations GPUDirect RDMA Direct receives used GDRDMA; sends used host staging. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read disabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:1 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce1; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce1; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-hybrid-r1-20260814` Task other Status VALID Measured 2026-08-14T15:43:39Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nwr0 Evidence distributed/results/20260814T154150Z-hybrid-rail1 Benchmark notes. Correct output; receive used GDRDMA and send used host staging. Result notes. Second single-port direct-receive control.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 fused ports; direct receive and staged send	valid	5	27.99	4.90	—	—	—	—	distributed/results/20260814T154150Z-hybrid-dual
`nw2` Two-rank FP16 all-reduce with direct receives on fused ports.1 result · 2 techniques · 13 parameters Configuration ID nccl-v100-p40-dual-hybrid-dual-fp16 Created 2026-08-14T15:44:02Z Notes NCCL_NET_GDR_READ=0; both HCA ports had nonzero counters. Applied optimizations GPUDirect RDMA Direct receives used GDRDMA on both fused ports. Multi-rail transport Both ports carried data and raised 16 MiB bus bandwidth by 13.7%. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read disabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:1,mlx4_0:2 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-hybrid-dual-20260814` Task other Status VALID Measured 2026-08-14T15:44:02Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nwr0 Evidence distributed/results/20260814T154150Z-hybrid-dual Benchmark notes. Both ports carried a 50/50 transmit split in every valid sample. Result notes. Dual rail raised 16 MiB bus bandwidth by 13.7%.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 port 2; full GDR	valid	5	32.26	2.31	—	—	—	—	distributed/results/20260814T154150Z-full-rail0
`nfr0` Two-rank FP16 all-reduce with full GDR on port 2.1 result · 1 technique · 13 parameters Configuration ID nccl-v100-p40-dual-full-r0-fp16 Created 2026-08-14T15:44:24Z Notes NCCL_NET_GDR_READ=1. Applied optimizations GPUDirect RDMA Sends and receives used GDRDMA. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read enabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:2 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-full-r0-20260814` Task other Status VALID Measured 2026-08-14T15:44:24Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nhr0 Evidence distributed/results/20260814T154150Z-full-rail0 Benchmark notes. Correct output and GDRDMA send and receive markers. Result notes. Single-port full-GDR control.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 port 1; full GDR	valid	5	21.60	2.31	—	—	—	—	distributed/results/20260814T154150Z-full-rail1
`nfr1` Two-rank FP16 all-reduce with full GDR on port 1.1 result · 1 technique · 13 parameters Configuration ID nccl-v100-p40-dual-full-r1-fp16 Created 2026-08-14T15:44:49Z Notes NCCL_NET_GDR_READ=1. Applied optimizations GPUDirect RDMA Sends and receives used GDRDMA. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read enabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:1 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce1; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce1; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-full-r1-20260814` Task other Status VALID Measured 2026-08-14T15:44:49Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nfr0 Evidence distributed/results/20260814T154150Z-full-rail1 Benchmark notes. Correct output and GDRDMA send and receive markers. Result notes. Second single-port full-GDR control.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 fused ports; full GDR	valid	5	29.20	2.30	—	—	—	—	distributed/results/20260814T154150Z-full-dual
`nf2` Two-rank FP16 all-reduce with full GDR on fused ports.1 result · 2 techniques · 13 parameters Configuration ID nccl-v100-p40-dual-full-dual-fp16 Created 2026-08-14T15:45:14Z Notes NCCL_NET_GDR_READ=1; both HCA ports had nonzero counters. Applied optimizations GPUDirect RDMA Sends and receives used GDRDMA on both fused ports. Multi-rail transport Both ports carried data, but 16 MiB bandwidth did not improve. Uncommon parameters backend.gdr enabled-phb — NCCL_NET_GDR_LEVEL was LOC or PHB. backend.gdr_read enabled — Hybrid mode set NCCL_NET_GDR_READ=0. network.control_path roce0;10.42.0.0/30 — NCCL socket and PRRTE control traffic stayed on rail 0. network.cross_nic 0 — NCCL_CROSS_NIC=0. network.gid_index 0 — RoCE v1 link-local GID. network.hca_ports mlx4_0:1,mlx4_0:2 ports — NCCL_IB_HCA selected the exact port list. network.merge_nics 1 — NCCL_IB_MERGE_NICS=1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.gpu_service_state lab-gpu-vram-eaters.target stopped — The target and its services were restored after the matrix. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.collective all-reduce — FP16 sum with two ranks. workload.iterations 50 iterations — Ten warm-up iterations; five independent samples. workload.message_range 4096-16777216 bytes — Power-of-two sweep. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Physical GPU 1; one MPI/NCCL rank. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Physical GPU 0; one MPI/NCCL rank. Results and evidence `nccl-v100-p40-dual-full-dual-20260814` Task other Status VALID Measured 2026-08-14T15:45:14Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline nfr0 Evidence distributed/results/20260814T154150Z-full-dual Benchmark notes. Both ports carried a 50/50 transmit split in every valid sample. Result notes. Dual rail did not improve 16 MiB full GDR.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair; NCCL selected SHM/direct/direct	valid	5	20.62	7.20	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`nl2i` NCCL all-reduce over both local V100s under interleave-all.1 result · 2 techniques · 0 parameters Configuration ID nccl-local-2v100-allreduce-interleave Created 2026-08-21T23:38:25Z Notes Two ranks in one process, FP16, 50 iterations after 10 warm iterations, with the nccl-tests correctness check enabled. Applied optimizations GPU peer access NCCL selected SHM/direct/direct; isAllCudaP2p was 0. Tensor parallelism The collective a tensor-parallel backend issues, measured without a model. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Rank 0. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; Rank 1. Results and evidence `nccl-local-2v100-allreduce-interleave-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.037 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. Two ranks in one process on one host. The transport line is retained in nccl-allreduce-interleave-transport.txt. Result notes. Seconds for one 256 MiB all-reduce, median of five samples. Sorted bus bandwidths: 7.17, 7.20, 7.21, 7.24, and 7.25 GB/s. Every sample passed the nccl-tests correctness check with zero out-of-bounds values.
	NCCL Tests FP16 all-reduce	NCCL Tests	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 pair; NCCL selected SHM/direct/direct	valid	5	20.70	7.01	—	—	—	—	pcie/results/20260821T233825Z-r720-nccl-p2p
`nl2n` NCCL all-reduce over both local V100s under NUMA node 0.1 result · 2 techniques · 0 parameters Configuration ID nccl-local-2v100-allreduce-node0 Created 2026-08-21T23:38:25Z Notes Two ranks in one process, FP16, 50 iterations after 10 warm iterations, with the nccl-tests correctness check enabled. Applied optimizations GPU peer access NCCL selected SHM/direct/direct; isAllCudaP2p was 0. Tensor parallelism The collective a tensor-parallel backend issues, measured without a model. Uncommon parameters No uncommon parameters are cataloged. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Rank 0. r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; Rank 1. Results and evidence `nccl-local-2v100-allreduce-node0-20260821` Task other Status VALID Measured 2026-08-21T23:38:25Z Statistic median; 5 samples Model NCCL Tests FP16 all-reduce Backend NCCL Tests Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.038 s Output matches baseline yes Evidence pcie/results/20260821T233825Z-r720-nccl-p2p Benchmark notes. Two ranks in one process on one host. The transport line is retained in nccl-allreduce-node0-transport.txt. Result notes. Seconds for one 256 MiB all-reduce, median of five samples. Sorted bus bandwidths: 7.11, 7.13, 7.14, 7.15, and 7.19 GB/s. Every sample passed the nccl-tests correctness check with zero out-of-bounds values.
	Network transport bandwidth preflight	iperf3	r720,r720xd	—	TCP over ConnectX-3 port 2	valid	5	—	—	—	—	—	—	distributed/results/20260814T153301Z-dual-rail-preflight
`dtcp0` One 10-second TCP stream on ConnectX-3 port 2.1 result · 0 techniques · 5 parameters Configuration ID connectx3-tcp-rail0-10s Created 2026-08-14T15:33:01Z Notes roce0 between 10.42.0.1 and 10.42.0.2. Applied optimizations No applied technique is cataloged. Uncommon parameters network.interfaces roce0 ports — mlx4_0:2. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.duration 10 seconds — One TCP stream per selected port. workload.samples 5 samples Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Server endpoint. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Client endpoint. Results and evidence `connectx3-tcp-rail0-20260814` Task other Status VALID Measured 2026-08-14T15:33:01Z Statistic median; 5 samples Model Network transport bandwidth preflight Backend iperf3 Hosts r720,r720xd Hardware CPU Evidence distributed/results/20260814T153301Z-dual-rail-preflight Benchmark notes. Median of five ten-second receive reports. Result notes. Single-port TCP control.
	Network transport bandwidth preflight	iperf3	r720,r720xd	—	TCP over ConnectX-3 port 1	valid	5	—	—	—	—	—	—	distributed/results/20260814T153301Z-dual-rail-preflight
`dtcp1` One 10-second TCP stream on ConnectX-3 port 1.1 result · 0 techniques · 5 parameters Configuration ID connectx3-tcp-rail1-10s Created 2026-08-14T15:33:01Z Notes roce1 between 10.42.0.5 and 10.42.0.6. Applied optimizations No applied technique is cataloged. Uncommon parameters network.interfaces roce1 ports — mlx4_0:1. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.duration 10 seconds — One TCP stream per selected port. workload.samples 5 samples Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce1; Server endpoint. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce1; Client endpoint. Results and evidence `connectx3-tcp-rail1-20260814` Task other Status VALID Measured 2026-08-14T15:33:01Z Statistic median; 5 samples Model Network transport bandwidth preflight Backend iperf3 Hosts r720,r720xd Hardware CPU Baseline dtcp0 Evidence distributed/results/20260814T153301Z-dual-rail-preflight Benchmark notes. Median of five ten-second receive reports. Result notes. The two single ports were within 2.9%.
	Network transport bandwidth preflight	iperf3	r720,r720xd	—	Concurrent TCP over both ConnectX-3 ports	valid	5	—	—	—	—	—	—	distributed/results/20260814T153301Z-dual-rail-preflight
`dtcp2` Two concurrent 10-second TCP streams on both ConnectX-3 ports.1 result · 0 techniques · 5 parameters Configuration ID connectx3-tcp-dual-10s Created 2026-08-14T15:33:01Z Notes Aggregate of the matching sample pair. Applied optimizations No applied technique is cataloged. Uncommon parameters network.interfaces roce0,roce1 ports — Two concurrent streams. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.duration 10 seconds — One TCP stream per selected port. workload.samples 5 samples Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Server endpoint. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Client endpoint. Results and evidence `connectx3-tcp-dual-20260814` Task other Status VALID Measured 2026-08-14T15:33:01Z Statistic median; 5 samples Model Network transport bandwidth preflight Backend iperf3 Hosts r720,r720xd Hardware CPU Baseline dtcp0 Evidence distributed/results/20260814T153301Z-dual-rail-preflight Benchmark notes. Median of five matching-sample aggregate receive rates. Result notes. Concurrent aggregate was 47.9% above the rail 0 median.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:05:00.0; NUMA node 0	valid	5	—	—	—	—	2.13	2.06	pcie/results/20260821T104000Z-r720-pcie-x16
`px0ia` CUDA0 bulk copy from pageable memory under interleave-all.1 result · 2 techniques · 5 parameters Configuration ID pcie-xfer-cuda0-interleave-pageable-8gib Created 2026-08-21T17:36:23Z Notes The combination a llama.cpp serving process performs today. Applied optimizations GPU link topology The combination a serving process performs today. NUMA memory policy numactl --interleave=all cost 35.6% of the pageable rate. Uncommon parameters accelerator:r720-v100-32gb.pcie_link PCIe 3.0 x16 at 0000:05:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pageable — malloc memory that the driver stages through its own pinned buffer. system.host_memory_policy numactl --interleave=all — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; One process copying to and from CUDA device 0. Results and evidence `r720-pcie-cuda0-interleave-pageable-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 4.037 s Baseline px1ia Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pageable host memory under interleave-all. The link that changed on 2026-08-21. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 2.04, 2.06, 2.13, 2.13, and 2.14 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:05:00.0; NUMA node 0	valid	5	—	—	—	—	11.43	12.48	pcie/results/20260821T104000Z-r720-pcie-x16
`px0ip` CUDA0 bulk copy from pinned memory under interleave-all.1 result · 3 techniques · 5 parameters Configuration ID pcie-xfer-cuda0-interleave-pinned-8gib Created 2026-08-21T17:36:23Z Notes The memory policy the serving profiles run under. Applied optimizations GPU link topology Same link under the serving memory policy. NUMA memory policy numactl --interleave=all cost 3.1% of the pinned rate. Pinned host transfer buffers Page-locked buffers under interleave-all. Uncommon parameters accelerator:r720-v100-32gb.pcie_link PCIe 3.0 x16 at 0000:05:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pinned — cudaHostAlloc page-locked memory. system.host_memory_policy numactl --interleave=all — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; One process copying to and from CUDA device 0. Results and evidence `r720-pcie-cuda0-interleave-pinned-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.752 s Baseline px1ip Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pinned host memory under interleave-all. The link that changed on 2026-08-21. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 11.35, 11.43, 11.43, 11.50, and 11.53 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:05:00.0; NUMA node 0	valid	5	—	—	—	—	3.31	3.24	pcie/results/20260821T104000Z-r720-pcie-x16
`px0la` CUDA0 bulk copy from pageable memory on its own NUMA node.1 result · 2 techniques · 5 parameters Configuration ID pcie-xfer-cuda0-local-pageable-8gib Created 2026-08-21T17:36:23Z Notes The driver stages this copy through an internal pinned buffer. Applied optimizations GPU link topology The link is idle here; the staging memcpy is the limit. NUMA memory policy Pageable staging on the card own NUMA node. Uncommon parameters accelerator:r720-v100-32gb.pcie_link PCIe 3.0 x16 at 0000:05:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pageable — malloc memory that the driver stages through its own pinned buffer. system.host_memory_policy numactl --cpunodebind=0 --membind=0 — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; One process copying to and from CUDA device 0. Results and evidence `r720-pcie-cuda0-local-pageable-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 2.593 s Baseline px1la Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pageable host memory under own NUMA node 0. The link that changed on 2026-08-21. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 3.02, 3.30, 3.31, 3.32, and 3.32 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:05:00.0; NUMA node 0	valid	5	—	—	—	—	11.79	12.95	pcie/results/20260821T104000Z-r720-pcie-x16
`px0lp` CUDA0 bulk copy from pinned memory on its own NUMA node.1 result · 3 techniques · 5 parameters Configuration ID pcie-xfer-cuda0-local-pinned-8gib Created 2026-08-21T17:36:23Z Notes Best case for the x16 link at 0000:05:00.0. Applied optimizations GPU link topology Measures the x16 link at 0000:05:00.0 that replaced an x8 link. NUMA memory policy Host memory bound to the card own NUMA node. Pinned host transfer buffers Page-locked source and sink buffers. Uncommon parameters accelerator:r720-v100-32gb.pcie_link PCIe 3.0 x16 at 0000:05:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pinned — cudaHostAlloc page-locked memory. system.host_memory_policy numactl --cpunodebind=0 --membind=0 — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 0; One process copying to and from CUDA device 0. Results and evidence `r720-pcie-cuda0-local-pinned-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.728 s Baseline px1lp Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pinned host memory under own NUMA node 0. The link that changed on 2026-08-21. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 11.76, 11.78, 11.79, 11.81, and 11.85 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:42:00.0; NUMA node 1	valid	5	—	—	—	—	2.01	1.94	pcie/results/20260821T104000Z-r720-pcie-x16
`px1ia` CUDA1 bulk copy from pageable memory under interleave-all.1 result · 2 techniques · 5 parameters Configuration ID pcie-xfer-cuda1-interleave-pageable-8gib Created 2026-08-21T17:36:23Z Notes Unchanged x16 control for the serving copy path. Applied optimizations GPU link topology Unchanged control for the serving copy path. NUMA memory policy numactl --interleave=all cost 41.6% of the pageable rate. Uncommon parameters accelerator:r720-v100-32gb-2.pcie_link PCIe 3.0 x16 at 0000:42:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pageable — malloc memory that the driver stages through its own pinned buffer. system.host_memory_policy numactl --interleave=all — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; One process copying to and from CUDA device 1. Results and evidence `r720-pcie-cuda1-interleave-pageable-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 4.273 s Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pageable host memory under interleave-all. Unchanged x16 control. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 1.98, 2.01, 2.01, 2.09, and 2.12 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:42:00.0; NUMA node 1	valid	5	—	—	—	—	11.41	12.56	pcie/results/20260821T104000Z-r720-pcie-x16
`px1ip` CUDA1 bulk copy from pinned memory under interleave-all.1 result · 3 techniques · 5 parameters Configuration ID pcie-xfer-cuda1-interleave-pinned-8gib Created 2026-08-21T17:36:23Z Notes Unchanged x16 control under the serving memory policy. Applied optimizations GPU link topology Unchanged control under the serving memory policy. NUMA memory policy numactl --interleave=all cost 5.7% of the pinned rate. Pinned host transfer buffers Page-locked buffers under interleave-all. Uncommon parameters accelerator:r720-v100-32gb-2.pcie_link PCIe 3.0 x16 at 0000:42:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pinned — cudaHostAlloc page-locked memory. system.host_memory_policy numactl --interleave=all — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA interleave all; One process copying to and from CUDA device 1. Results and evidence `r720-pcie-cuda1-interleave-pinned-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.753 s Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pinned host memory under interleave-all. Unchanged x16 control. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 11.35, 11.37, 11.41, 11.47, and 11.53 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:42:00.0; NUMA node 1	valid	5	—	—	—	—	3.44	3.26	pcie/results/20260821T104000Z-r720-pcie-x16
`px1la` CUDA1 bulk copy from pageable memory on its own NUMA node.1 result · 2 techniques · 5 parameters Configuration ID pcie-xfer-cuda1-local-pageable-8gib Created 2026-08-21T17:36:23Z Notes Unchanged x16 control for the staged copy path. Applied optimizations GPU link topology Unchanged control for the staged copy path. NUMA memory policy Pageable staging on the card own NUMA node. Uncommon parameters accelerator:r720-v100-32gb-2.pcie_link PCIe 3.0 x16 at 0000:42:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pageable — malloc memory that the driver stages through its own pinned buffer. system.host_memory_policy numactl --cpunodebind=1 --membind=1 — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 1; One process copying to and from CUDA device 1. Results and evidence `r720-pcie-cuda1-local-pageable-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 2.498 s Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pageable host memory under own NUMA node 1. Unchanged x16 control. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 3.03, 3.15, 3.44, 3.48, and 3.49 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	PCIe host-device transfer	host-device-xfer	r720	Tesla PG500-216 (V100 32GB)	PCIe 3.0 x16 at 0000:42:00.0; NUMA node 1	valid	5	—	—	—	—	12.10	13.01	pcie/results/20260821T104000Z-r720-pcie-x16
`px1lp` CUDA1 bulk copy from pinned memory on its own NUMA node.1 result · 3 techniques · 5 parameters Configuration ID pcie-xfer-cuda1-local-pinned-8gib Created 2026-08-21T17:36:23Z Notes Unchanged x16 control at 0000:42:00.0. Applied optimizations GPU link topology Unchanged x16 control at 0000:42:00.0. NUMA memory policy Host memory bound to the card own NUMA node. Pinned host transfer buffers Page-locked source and sink buffers. Uncommon parameters accelerator:r720-v100-32gb-2.pcie_link PCIe 3.0 x16 at 0000:42:00.0 lanes — sysfs current_link_width and max_link_width both report 16. system.host_memory_kind pinned — cudaHostAlloc page-locked memory. system.host_memory_policy numactl --cpunodebind=1 --membind=1 — Host allocation and thread placement for the copying process. system.resident_gpu_services none bytes — lab-gpu-vram-eaters.target was stopped and both cards reported 0 MiB in use before the first copy. system.transfer_bytes 8589934592 bytes — One session of the measured agent workload holds 8,739 MiB. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA cpunodebind and membind 1; One process copying to and from CUDA device 1. Results and evidence `r720-pcie-cuda1-local-pinned-20260821` Task other Status VALID Measured 2026-08-21T17:36:23Z Statistic median; 5 samples Model PCIe host-device transfer Backend host-device-xfer Hosts r720 Hardware Tesla PG500-216 (V100 32GB) Elapsed 0.710 s Output matches baseline yes Evidence pcie/results/20260821T104000Z-r720-pcie-x16 Benchmark notes. Pinned host memory under own NUMA node 1. Unchanged x16 control. Result notes. Host-to-device seconds for 8,192 MiB, median of five samples. Sorted rates: 12.09, 12.10, 12.10, 12.10, and 12.14 GB/s. Every arm copied a position-dependent pattern to the device and back and compared the bytes.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	—	RoCE v1 RDMA WRITE; host to host	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-write-host-host
`rdwhh` 8 MiB RDMA WRITE from P40-host memory to V100-host memory.1 result · 0 techniques · 4 parameters Configuration ID rdma-write-host-host-8m Created 2026-08-10T23:05:38Z Notes Five samples; 100 operations per sample. Applied optimizations No applied technique is cataloged. Uncommon parameters workload.destination_memory host workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory host workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory destination. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory source. Results and evidence `rdma-write-host-host-20260810` Task other Status VALID Measured 2026-08-10T23:05:38Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware CPU Evidence distributed/results/20260810T225836Z-cuda-rdma-write-host-host Benchmark notes. Host-memory bandwidth control. Result notes. Sorted samples: 4640.20, 4653.22, 4654.67, 4656.56, and 4656.63 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	Tesla PG500-216 (V100 32GB)	RoCE v1 RDMA WRITE; host to V100 CUDA memory	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-write-host-to-v100
`rdwhv` 8 MiB RDMA WRITE from P40-host memory into V100 device memory.1 result · 1 technique · 4 parameters Configuration ID rdma-write-host-v100-8m Created 2026-08-10T23:06:00Z Notes Direct NIC write into the V100. Applied optimizations GPUDirect RDMA ConnectX-3 wrote directly into registered V100 device memory. Uncommon parameters workload.destination_memory v100-cuda — Physical GPU 1 on r720. workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory host workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; V100 CUDA-memory destination. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory source. Results and evidence `rdma-write-host-v100-20260810` Task other Status VALID Measured 2026-08-10T23:06:00Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB) Baseline rdwhh Evidence distributed/results/20260810T225836Z-cuda-rdma-write-host-to-v100 Benchmark notes. Every server sample allocated and registered a V100 CUDA buffer. Result notes. Sorted samples: 4626.00, 4656.85, 4657.05, 4657.63, and 4658.27 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	Tesla P40	RoCE v1 RDMA WRITE; P40 CUDA memory to host	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-write-p40-to-host
`rdwph` 8 MiB RDMA WRITE from P40 device memory to V100-host memory.1 result · 1 technique · 4 parameters Configuration ID rdma-write-p40-host-8m Created 2026-08-10T23:06:25Z Notes Direct NIC read from the P40. Applied optimizations GPUDirect RDMA ConnectX-3 read directly from registered P40 device memory. Uncommon parameters workload.destination_memory host workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory p40-cuda — Physical GPU 0 on r720xd. workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory destination. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; P40 CUDA-memory source. Results and evidence `rdma-write-p40-host-20260810` Task other Status VALID Measured 2026-08-10T23:06:25Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware Tesla P40 Baseline rdwhh Evidence distributed/results/20260810T225836Z-cuda-rdma-write-p40-to-host Benchmark notes. Every client sample allocated and registered a P40 CUDA buffer. Result notes. Sorted samples: 2925.62, 2926.76, 2927.83, 2930.13, and 2934.08 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 RDMA WRITE; P40 CUDA memory to V100 CUDA memory	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-write-p40-to-v100
`rdwpv` 8 MiB RDMA WRITE from P40 device memory into V100 device memory.1 result · 1 technique · 4 parameters Configuration ID rdma-write-p40-v100-8m Created 2026-08-10T23:06:50Z Notes Direct read from P40 and write into V100. Applied optimizations GPUDirect RDMA Both RDMA endpoints used registered CUDA device memory. Uncommon parameters workload.destination_memory v100-cuda — Physical GPU 1 on r720. workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory p40-cuda — Physical GPU 0 on r720xd. workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; V100 CUDA-memory destination. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; P40 CUDA-memory source. Results and evidence `rdma-write-p40-v100-20260810` Task other Status VALID Measured 2026-08-10T23:06:50Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline rdwhh Evidence distributed/results/20260810T225836Z-cuda-rdma-write-p40-to-v100 Benchmark notes. Both endpoints allocated and registered CUDA device buffers. Result notes. Sorted samples: 2920.91, 2924.55, 2926.08, 2926.87, and 2927.93 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	—	RoCE v1 RDMA READ; host to host	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-read-host-host
`rdrhh` 8 MiB RDMA READ from V100-host memory into P40-host memory.1 result · 0 techniques · 4 parameters Configuration ID rdma-read-host-host-8m Created 2026-08-10T23:07:14Z Notes Five samples; 100 operations per sample. Applied optimizations No applied technique is cataloged. Uncommon parameters workload.destination_memory host workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory host workload.verb rdma-read — Server r720 was the source; client r720xd was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory source. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory destination. Results and evidence `rdma-read-host-host-20260810` Task other Status VALID Measured 2026-08-10T23:07:14Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware CPU Evidence distributed/results/20260810T225836Z-cuda-rdma-read-host-host Benchmark notes. Host-memory bandwidth control. Result notes. Sorted samples: 4643.93, 4656.67, 4657.68, 4657.95, and 4658.15 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	Tesla PG500-216 (V100 32GB)	RoCE v1 RDMA READ; V100 CUDA memory to host	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-read-v100-to-host
`rdrvh` 8 MiB RDMA READ from V100 device memory into P40-host memory.1 result · 1 technique · 4 parameters Configuration ID rdma-read-v100-host-8m Created 2026-08-10T23:07:33Z Notes Direct NIC read from the V100. Applied optimizations GPUDirect RDMA ConnectX-3 read directly from registered V100 device memory. Uncommon parameters workload.destination_memory host workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory v100-cuda — Physical GPU 1 on r720. workload.verb rdma-read — Server r720 was the source; client r720xd was the destination. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; V100 CUDA-memory source. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory destination. Results and evidence `rdma-read-v100-host-20260810` Task other Status VALID Measured 2026-08-10T23:07:33Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB) Baseline rdrhh Evidence distributed/results/20260810T225836Z-cuda-rdma-read-v100-to-host Benchmark notes. Every server sample allocated and registered a V100 CUDA buffer. Result notes. Sorted samples: 2447.82, 2450.25, 2451.20, 2451.53, and 2456.57 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	Tesla P40	RoCE v1 RDMA READ; host to P40 CUDA memory	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-read-host-to-p40
`rdrhp` 8 MiB RDMA READ from V100-host memory into P40 device memory.1 result · 1 technique · 4 parameters Configuration ID rdma-read-host-p40-8m Created 2026-08-10T23:07:57Z Notes Direct NIC write into the P40. Applied optimizations GPUDirect RDMA ConnectX-3 wrote directly into registered P40 device memory. Uncommon parameters workload.destination_memory p40-cuda — Physical GPU 0 on r720xd. workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory host workload.verb rdma-read — Server r720 was the source; client r720xd was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Host-memory source. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; P40 CUDA-memory destination. Results and evidence `rdma-read-host-p40-20260810` Task other Status VALID Measured 2026-08-10T23:07:57Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware Tesla P40 Baseline rdrhh Evidence distributed/results/20260810T225836Z-cuda-rdma-read-host-to-p40 Benchmark notes. Every client sample allocated and registered a P40 CUDA buffer. Result notes. Sorted samples: 4650.34, 4655.81, 4656.46, 4657.12, and 4658.00 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	Tesla PG500-216 (V100 32GB),Tesla P40	RoCE v1 RDMA READ; V100 CUDA memory to P40 CUDA memory	valid	5	—	—	—	—	—	—	distributed/results/20260810T225836Z-cuda-rdma-read-v100-to-p40
`rdrvp` 8 MiB RDMA READ from V100 device memory into P40 device memory.1 result · 1 technique · 4 parameters Configuration ID rdma-read-v100-p40-8m Created 2026-08-10T23:08:19Z Notes Direct read from V100 and write into P40. Applied optimizations GPUDirect RDMA Both RDMA endpoints used registered CUDA device memory. Uncommon parameters workload.destination_memory p40-cuda — Physical GPU 0 on r720xd. workload.message_size 8388608 bytes — One hundred operations per sample. workload.source_memory v100-cuda — Physical GPU 1 on r720. workload.verb rdma-read — Server r720 was the source; client r720xd was the destination. Node placement r720 — inference; CUDA; Tesla PG500-216 (V100 32GB); NUMA NUMA node 1; PHB-local NIC; network roce0; V100 CUDA-memory source. r720xd — inference; CUDA; Tesla P40; NUMA NUMA node 1; PHB-local NIC; network roce0; P40 CUDA-memory destination. Results and evidence `rdma-read-v100-p40-20260810` Task other Status VALID Measured 2026-08-10T23:08:19Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware Tesla PG500-216 (V100 32GB),Tesla P40 Baseline rdrhh Evidence distributed/results/20260810T225836Z-cuda-rdma-read-v100-to-p40 Benchmark notes. Both endpoints allocated and registered CUDA device buffers. Result notes. Sorted samples: 2456.91, 2457.53, 2460.44, 2460.52, and 2461.98 MiB/s.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	—	RoCE v1 RDMA WRITE over ConnectX-3 port 2	valid	5	—	—	—	—	—	—	distributed/results/20260814T153301Z-verbs-host-rail0
`drdma0` 8 MiB host-memory RDMA WRITE on ConnectX-3 port 2.1 result · 0 techniques · 7 parameters Configuration ID rdma-write-host-host-8m-rail0 Created 2026-08-14T15:37:54Z Notes One hundred operations per sample. Applied optimizations No applied technique is cataloged. Uncommon parameters network.active_mtu 4096 bytes — Ethernet MTU was 9000. network.gid_index 0 network.hca_ports mlx4_0:2 ports system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.message_size 8388608 bytes — One hundred operations per sample. workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Server endpoint. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0; Client endpoint. Results and evidence `rdma-write-host-host-rail0-20260814` Task other Status VALID Measured 2026-08-14T15:37:54Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware CPU Evidence distributed/results/20260814T153301Z-verbs-host-rail0 Benchmark notes. Median of five client reports. Result notes. Single-port verbs control.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	—	RoCE v1 RDMA WRITE over ConnectX-3 port 1	valid	5	—	—	—	—	—	—	distributed/results/20260814T153301Z-verbs-host-rail1
`drdma1` 8 MiB host-memory RDMA WRITE on ConnectX-3 port 1.1 result · 0 techniques · 7 parameters Configuration ID rdma-write-host-host-8m-rail1 Created 2026-08-14T15:38:18Z Notes One hundred operations per sample. Applied optimizations No applied technique is cataloged. Uncommon parameters network.active_mtu 4096 bytes — Ethernet MTU was 9000. network.gid_index 0 network.hca_ports mlx4_0:1 ports system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.message_size 8388608 bytes — One hundred operations per sample. workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce1; Server endpoint. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce1; Client endpoint. Results and evidence `rdma-write-host-host-rail1-20260814` Task other Status VALID Measured 2026-08-14T15:38:18Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware CPU Baseline drdma0 Evidence distributed/results/20260814T153301Z-verbs-host-rail1 Benchmark notes. Median of five client reports. Result notes. The two single ports were within 0.1%.
	perftest CUDA RDMA bandwidth	perftest	r720,r720xd	—	Concurrent RoCE v1 RDMA WRITEs over both ports	valid	5	—	—	—	—	—	—	distributed/results/20260814T154150Z-dual-rail-roce.md
`drdma2` Concurrent 8 MiB host-memory RDMA WRITEs on both ConnectX-3 ports.1 result · 1 technique · 7 parameters Configuration ID rdma-write-host-host-8m-dual Created 2026-08-14T15:38:46Z Notes Aggregate of the matching sample pair. Applied optimizations Multi-rail transport Two concurrent port processes raised aggregate bandwidth by 9.6%. Uncommon parameters network.active_mtu 4096 bytes — Ethernet MTU was 9000. network.gid_index 0 network.hca_ports mlx4_0:1,mlx4_0:2 ports — Two concurrent processes. system.benchmark_harness_revision 178affcb72afcf9ee32fdb22b7a0eb0b57bfc52c — Tracked local-ai revision for every retained point. system.network_configuration_revision e509c5d — Deployed Bench revision with both persistent RoCE interfaces. workload.message_size 8388608 bytes — One hundred operations per sample. workload.verb rdma-write — Client r720xd was the source; server r720 was the destination. Node placement r720 — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Server endpoint. r720xd — inference; CPU; NUMA NUMA node 1; PHB-local NIC; network roce0,roce1; Client endpoint. Results and evidence `rdma-write-host-host-dual-20260814` Task other Status VALID Measured 2026-08-14T15:38:46Z Statistic median; 5 samples Model perftest CUDA RDMA bandwidth Backend perftest Hosts r720,r720xd Hardware CPU Baseline drdma0 Evidence distributed/results/20260814T154150Z-dual-rail-roce.md Benchmark notes. Median of five matching-sample aggregate bandwidth values. Result notes. Concurrent aggregate was 9.6% above the rail 0 median.

Benchmarks

Needle 2

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

needle2-bfclv4-planned-20260811

needle2-cpu-t16-throughput-planned-20260811

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

needle2-cpu-t1-throughput-planned-20260811

TinyLlama OpenOrca 1.1B

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

tinyllama-p40-rpc-control-20260811

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

tinyllama-v100-20260729

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

tinyllama-v100-control-20260811

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

tinyllama-v100-p40-tp-20260811

Bonsai

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

bonsai8-cpu-t16

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

bonsai8-p40

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

bonsai8-v100

Gemma 4

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

gemma26-cpu-f714bf0-20260726

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

gemma26-cpu-t12

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

gemma26-mtp-baseline

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

gemma26-asyncclient-baseline-smoke-20260728

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

gemma26-mtp-f714bf0-w1-20260726

Applied optimizations

Uncommon parameters

Node placement

Results and evidence

`needle2-bfclv4-planned-20260811`

`needle2-cpu-t16-throughput-planned-20260811`

`needle2-cpu-t1-throughput-planned-20260811`

`tinyllama-p40-rpc-control-20260811`

`tinyllama-v100-20260729`

`tinyllama-v100-control-20260811`

`tinyllama-v100-p40-tp-20260811`

`bonsai8-cpu-t16`

`bonsai8-p40`

`bonsai8-v100`

`gemma26-cpu-f714bf0-20260726`

`gemma26-cpu-t12`

`gemma26-mtp-baseline`

`gemma26-asyncclient-baseline-smoke-20260728`

`gemma26-mtp-f714bf0-w1-20260726`

`gemma26-mtp-w1`

`gemma26-r720-full-remote-w1-smoke-20260728`

`gemma26-v100`

`gemma26-v100-mtp-baseline`

`gemma26-v100-mtp-w1`

`qwen38-2xv100-layer-baseline-20260814`

`qwen38-2xv100-replicas-20260814`

`qwen38-b10358-single-off-code-20260819`

`qwen38-b10358-single-off-prose-20260819`

`qwen38-b10488-single-off-cuda1-code-20260819`

`qwen38-b10488-single-off-cuda1-prose-20260819`

`qwen38-b10488-single-off-nograph-code-20260819`

`qwen38-b10488-single-off-nograph-prose-20260819`

`qwen38-b10488-single-off-code-20260819`

`qwen38-b10488-single-off-prose-20260819`

`qwen38-b9190-single-off-code-20260819`

`qwen38-b9190-single-off-prose-20260819`