Sloptimized

Reusable techniques and measured applications

Optimization catalogue.

98% agent-driven model benchmarks and optimizations.

Optimizations

The catalog connects reusable optimization techniques to measured runs. The change column reports decode throughput against the named baseline configuration. A row can include several techniques, so the change does not isolate one technique's effect. Expand a row to see what changed, the limits of the result, and the retained evidence.

Standout improvements

Largest valid gains with recorded baseline output agreement.

+525.4%

Mistral Small 4 119B 2603

ms4d reached 75.1 decode tok/s versus ms4s.

+257.9%

GLM-5.2

col2g reached 0.6 decode tok/s versus colq1.

+218.4%

Qwen2.5 72B Instruct

q72d2 reached 8.3 decode tok/s versus q72s2.

+79.7%

GLM-5.2

r4fv0 reached 0.7 decode tok/s versus r4fs0.

Techniques and rows are ordered by improvement. Each section gives the generic guidance. Select a configuration row to show its application notes and retained evidence.

GPU layer offload

Best associated +951.0%

Place target-model tensors in accelerator memory.

General guidance. GPU memory usually gives more bandwidth than host memory. Increase the offloaded target layers until the model, KV cache, compute graphs, and service reserve fit. Use matched benchmarks because a placement change also changes the CPU and GPU work balance. Check model buffer locations as well as the offloaded layer count. The Qwen3.5 Q5 four-V100 control on b10488 reported 50/50 layers offloaded but retained 500.16 MiB in CUDA_Host. A complete layer count alone does not prove that all model tensors are on GPUs.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

GPU peer access

Best associated +951.0%

Use direct transfers between accelerators when the platform permits.

General guidance. Direct peer access can remove a host-memory copy from multi-GPU traffic. Topology, firmware, the operating system, and IOMMU policy can block or slow this path. Test bidirectional bandwidth and latency before and after peer access is enabled.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Same-host layer split

Best associated +951.0%

Place different model layers on local accelerators.

General guidance. A layer split combines accelerator memory and transfers activations at layer boundaries. It is often a useful first multi-accelerator mode when the model does not fit on one device. Tune the split for memory balance and keep the full target on one host when possible. On the R720 PCIe SYS path without peer access, Mistral Small 4 row split was 53.1% slower for decode than layer split. A layer split earns nothing for a model that fits on one card. Qwen3.8 27B Q4_K_M is 15.9 GiB against 32 GiB per V100, and single placement matched the 1:1 split at every point measured, including 45.604 against 46.441 tok/s at a 30,906-token prompt. The split also made the speculative arm unstable: three of five layer-split MTP arms spread between 11.7% and 26.8% across their samples, while no single-card arm exceeded 2%.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Architecture-specific build

Best associated +289.5%

Build the runtime for the CPU and accelerator instruction sets.

General guidance. A common compatibility build can omit useful CPU instructions or accelerator code. Pin the source revision, compiler, and target architectures. A compiler function target can specialize one hot path without raising the instruction-set floor of the complete engine. Do not run a newer instruction-set path on an older host.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Kernel-path tuning

Best associated +289.5%

Select and tune kernels for the exact model and batch shapes.

General guidance. Automatic selection can use a slow fallback on an older accelerator or an uncommon tensor shape. Compare the supported matrix-multiply paths and exact-shape kernels. Record build flags, memory use, prefill speed, decode speed, and output checks. An exact grouped-int4 CPU path can vectorize independent output rows while preserving the scalar reduction order within each row. On dual Ivy Bridge E5-2680 v2 CPUs, Colibri PR 1286 measured exact kernel speedups of 2.153989 times at 20 threads and 2.655422 times at 40 threads. A matched five-sample streaming GLM-5.2 comparison measured a 1.796942-times median decode gain and 41.644% lower median TTFT, with identical text and canonical-token hashes. The candidate throughput range was wide, so retain and report the full range.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Weight quantization

Best associated +289.5%

Use lower-precision model weights to reduce storage and memory use.

General guidance. A smaller weight format can let more tensors stay in fast memory. The gain depends on hardware support, dequantization kernels, tensor shapes, and batch size. Compare task quality as well as speed. Do not transfer a result between different quantized artifacts. For CPU-resident weights on the AVX-only R720, ggml's dot product reads Q4_0, Q8_0, Q4_K to Q6_K, IQ4_NL, IQ4_XS, and MXFP4 at 89-98% of memory bandwidth and Q2_K at 83%, but Q3_K, IQ3, IQ2, IQ1, Q2_0, and Q1_0 at only 17-48%. Below 4 bits those weights save memory but not decode time until a faster kernel exists: IQ2_XXS computes 57 G weights/s per socket and Q4_K 66 G (r720-cpu-kernel-roofline-20260928). Judge a format's quality cost by its mean KL divergence from a float base (kl-screen), not by a perplexity ratio, whose standard error at 900 tokens is as large as the differences between formats; a faster format qualifies when its extra KL divergence in percent is at most 0.1 times its decode gain in percent, with one standard error of margin (OPTIMIZATION-GUIDE.md section 2). File size does not predict speed where decode is compute-bound: on the R720 CPU a 0.8B model decoded fastest in Q8_0 (26.6 tok/s) and slower in every K-quant (17.9 to 22.6 tok/s), so the rule's gain must come from a matched pair on the hardware that will serve the model (qwen35-0.8b-kl-screen).

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Asynchronous expert I/O

Best associated +257.9%

Overlap out-of-core expert reads with useful model computation.

General guidance. An out-of-core MoE runtime can issue missing-expert reads while resident experts execute. The gain depends on storage latency, worker count, CPU contention, cache residency, and how much useful matmul is available to hide the reads. Measure a long enough decode to reach the later expert working set; a short warm screen can substantially overstate steady throughput. Track felt wait separately from read-service time and retain enough warm samples for the buffered page cache to converge.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

MoE expert placement

Best associated +257.9%

Place routed-expert tensors independently from other model tensors.

General guidance. A sparse MoE model uses only some experts for each token. Keep high-value shared and non-expert tensors on the GPU, then tune how many expert blocks remain on the CPU. The best split depends on expert routing, memory bandwidth, context size, and free VRAM. A runtime with expert-level staging can extend the split across VRAM, host RAM, and storage. Record the hit rate, bytes fetched per token, and read-wait share because correct output alone does not show that the storage tier is fast enough for the workload. On multi-GPU hosts, confirm that the backend selected every intended device and release redundant host copies before using the result to size RAM. A remote accelerator can be the expert-capacity tier. Qwen3.5 122B-A10B Q5_K_M replaced a 25,554.57 MiB CUDA_Host model buffer with a 16-layer remote V100 share, and every retained three-card sample exceeded every fitted two-card CPU-offload sample.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

NUMA memory policy

Best associated +257.9%

Place CPU memory near the device that consumes it.

General guidance. CPU tensors, draft work, and transfers can use different NUMA paths. Compare local allocation with interleave for the selected workload. Record the memory policy, CPU set, device node, and memory bandwidth. A backend NUMA flag such as llama.cpp --numa distribute places threads. It does not set the allocation policy of a private no-mmap weight buffer. Wrap the process in numactl --interleave=all to spread that buffer across the memory controllers. On the R720 this raised Laguna S decode about 30%. Interleave needs comparable free pages on every node; an unbalanced node makes results vary between runs. For a mmap-only backend, drop the page cache once and load under numactl; the file pages fault interleaved and the placement persists warm. This raised Qwen 72B prima decode 47.8%. The gain is not universal: a CPU-only Gemma 4 26B-A4B profile measured within noise of its mapped control, so measure the policy for each workload instead of assuming the gain.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Release GPU capacity

Best associated +257.9%

Remove auxiliary work from the target accelerator.

General guidance. Other GPU services use memory and scheduling time that the target needs. Stop them or move them to another accelerator or host. This can release memory for weights, KV cache, compute graphs, and draft models. Record the service placement for each test.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Speculative decoding

Best associated +65.8%

Use a cheaper predictor to propose tokens for target verification.

General guidance. A draft model or an MTP head proposes tokens before the target verifies them. Net speed depends on acceptance, draft latency, verification cost, and synchronization. Record a matched target baseline, output agreement, accepted-length distribution, proposal policy, and draft placement. Correct draft metadata is required. Muse Glimmer 30B on one V100 showed that useful width is both target- and quantization-specific: width one improved the 17 GB artifact by 20.1 percent, while width two improved the dynamic artifact by 34.8 percent. Width 16 reduced speed for both, and wider proposals caused output divergence on the 17 GB artifact. Qwen3.8 27B refined the width rule on a dense target: width one gave 1.38x, width two 1.43x, widths three and four 1.31x, and width six 1.01x on one V100, while acceptance fell from 87.3% to 43.0%. A high acceptance rate at a wide setting does not indicate a gain. The same head gained more at depth, 1.64x at a 30,906-token prompt with 88.1% acceptance, and more with generated length, 1.36x at 128 tokens against 1.52x at 1,024. Measure a speculative head on a current backend build: the same arm measured 1.27x on llama.cpp b9190 and 1.43x on b10358.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Speculative policy tuning

Best associated +65.8%

Tune proposal depth, lookahead, and verification branching.

General guidance. More proposals help only while accepted tokens repay draft and verification cost. Sweep small depths and any parallel proposal count. Stop when acceptance length or end-to-end throughput falls. Use the same target, prompt, sampling, and output checks. A minimum draft probability is not a default. Gating Qwen3.8 27B at 0.60 with width four reached 1.26x against 1.31x ungated at the same width and 1.43x at width two, so the gate lost on this hardware even though it raised acceptance from 0.57 to 0.71.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Draft placement

Best associated +52.2%

Place speculative draft work away from the target bottleneck.

General guidance. A draft can run on the target device, another accelerator, a CPU, or a remote host. The best location depends on free memory, draft latency, target interference, and transfer delay. Token-level traffic can tolerate a slower network than tensor traffic.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Host page cache

Best associated +47.8%

Keep mapped model files in the operating-system page cache.

General guidance. Large system RAM can make model reloads much faster and can let processes share clean mapped pages. Separate cold and warm start tests. Use proportional set size and major page faults to distinguish shared cache from private resident memory.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Cross-host layer split

Best associated +46.8%

Place contiguous model layers on accelerators in different hosts.

General guidance. A cross-host layer split combines device memory while transferring activations only at layer boundaries. It is useful when the remote capacity replaces CPU weights or compute; it is not automatically useful when the model already fits locally. Tune the remote share to its compute and transport path instead of assigning equal shares. Laguna S over llama.cpp RPC TCP and a measured 39.395 Gb/s direct link improved from 32.1608 to 38.9215 tok/s with 21:21:7 layers, while an approximately equal three-card split fell to 26.3659. Automatic fit can replace a requested split; disable it for manual placement and verify both layer assignments and model-buffer sizes. Qwen3.5 122B-A10B Q5_K_M confirms the capacity case and shows that the useful share is model-dependent. Its two-card fit retained 25,554.57 MiB in CUDA_Host, while an equal 17:17:16 three-card split removed that host model buffer and reproduced medians of 28.0916 and 27.5660 tok/s. The two-card controls were variable, so no exact median speedup is claimed; the slowest three-card sample was still 1.603689 times the fastest control sample.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Share GPUs between independent evaluation workers

Best associated +40.5%

Use spare memory and execution gaps for concurrent unchanged inference batches.

General guidance. A GPU with a resident service is not necessarily full. Several independent evaluation workers can fit if their peak allocations, allocator reservations, and resident services fit together. Match exact inputs and outputs, hold per-device work fixed, record startup and warmup separately, and monitor total device memory and thermals. In one ARC screen, five workers on two 32 GiB V100 cards processed 500 fixed candidate executions in 56.21 seconds versus 79.00 seconds with two workers, a 1.405x warm throughput gain. They used about 20.4 and 28.9 GiB, with speech left resident. Extra startup/warmup erased the gain for this short overall test. This does not establish speech latency or full-evaluation speedup.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Batch-size tuning

Best associated +39.9%

Tune logical and physical batch sizes for the workload.

General guidance. Batch and microbatch sizes change memory use, prompt throughput, and decode scheduling. They can also select different backend kernels. Large values can help prompt processing. Smaller values can release memory for more GPU layers, cache, parallel slots, or a draft model. The direction differs by architecture: a dense partial offload preferred batch 128 to gain GPU layers, but a hybrid MoE prefill processes CPU-resident experts in microbatch-size chunks, and microbatch 1024 nearly doubled Laguna S 4K prefill. Decode was insensitive to the same change.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Output tensor placement

Best associated +39.9%

Keep frequently used input and output tensors in accelerator memory.

General guidance. A hybrid model can still move data for token embeddings or output projection on each step. Place these tensors on the accelerator when they fit. Measure transfer traffic, memory use, and decode speed with a matched control.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Runtime selection

Best associated +39.9%

Select a supported runtime for the hardware and workload.

General guidance. Runtime support, model format, quantization kernels, and serving features change the result. Pin a compatible version and isolate legacy stacks. Compare the same model quality and workload. A newer runtime or a listed quantizer does not guarantee hardware support.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

CPU affinity

Best associated +37.7%

Bind inference threads to a selected CPU set.

General guidance. CPU affinity can keep work on the intended sockets and memory paths. It is most useful when CPU tensors and GPU transfers share a NUMA system. Record the exact CPU mask and the NUMA policy.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Flash attention

Best associated +37.7%

Use the backend fused attention path.

General guidance. Flash attention can reduce attention memory traffic and temporary storage. Its benefit depends on the backend, GPU architecture, context length, KV-cache type, and batch shape. Confirm that the runtime uses a supported kernel. Validate output and memory use with a matched control. The smaller compute buffer is a second lever: in a VRAM-limited partial offload it can free space for a larger microbatch or another expert block, which can be worth more than the kernel speedup itself.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Private weight copy

Best associated +37.7%

Use private host-memory model weights instead of mapped files.

General guidance. A no-mmap load uses private host memory for CPU-resident weights. This can avoid mapped-file behavior during mixed CPU and GPU inference, but it increases resident host memory. Compare it with a matched mapped control. The private buffer follows the process NUMA memory policy. On a multi-socket host, pair it with numa-memory-policy interleave, or the allocation lands on one node and loses the other memory controllers.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

CPU thread tuning

Best associated +18.8%

Match inference threads to the useful physical CPU resources.

General guidance. Thread count controls memory pressure, synchronization, and NUMA traffic. Test physical-core counts and nearby values. Keep batch threads separate when the backend supports this control.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Ordered whole-batch parallel evaluation

Best associated +5.7%

Generate unchanged batches on separate GPUs and restore their global order before voting.

General guidance. Preserve prompt content and batch padding, retain batch indexes across pause and resume, and apply one seeded global vote after restoring result order. One ARC screen reproduced all 288 generations, votes and scores exactly. Two V100s reduced active generation span by 47.8 percent but reduced total time by only 5.4 percent on eight tasks because worker startup was a large share. Full evaluation scaling remains unmeasured.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Distributed target inference

Best associated -0.9%

Split target-model work across more than one host.

General guidance. A layer or pipeline split can add memory and compute capacity, but network transfer and synchronization can remove the gain. Token messages need much less bandwidth than tensor or KV-cache traffic. Compare the split with a one-host target. Record each node role, transfer volume, and the physical interconnect.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

KV-cache quantization

Best associated -4.5%

Use lower-precision key and value cache entries.

General guidance. A smaller KV cache can increase context length or parallel slots. It is not always faster and can change output quality. Compare F16, a supported 8-bit format such as FP8 or Q8, and Q4 as separate series with fixed prompts and sampling. Some parallel modes or attention kernels require a specific cache type. On the V100 with flash attention, Q8_0 KV decoded about 8% slower than F16 for Laguna S; quantized cache pays dequantization work and only wins when the freed memory buys capacity the profile needs. An agent serving profile is the clearest case of such a need: the freed memory becomes retained prefix slots, and one avoided full prefill repays a large decode loss. Judge the trade with prefix-caching enabled and with the measured concurrency, not on a single stream.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Tensor parallelism

Best associated -14.9%

Split model tensors across accelerators for each compute step.

General guidance. Tensor parallelism can reduce latency and combine device memory. It also needs frequent collective communication. Test it only after the peer and collective paths work. Record the split, cache type, fused collective options, and communication time. Runtime and model shape support are prerequisites: llama.cpp 9190 could not load its experimental tensor split for Mistral Small 4. A heterogeneous cross-host split can execute correctly while losing most throughput when generic RPC TCP carries the per-layer reductions. A CUDA/NCCL MLP proof on the V100/P40 pair matched its local reference while every channel used RoCE GDRDMA, although the heterogeneous two-rank step was still slower than one V100 executing both shards. Verify the actual transport and compare against a local execution control. On a SYS-connected pair without peer access, tensor split is a loss. One Qwen3.8 27B process decoded 32.965 tok/s under layer split and 31.932 tok/s under tensor split, with tensor samples between 18.568 and 33.856 tok/s, and native MTP fell to 26.131 tok/s under tensor split against 43.257 under layer split. Published tensor-split gains from cards under one host bridge do not transfer to a socket-crossing pair.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Bound file descriptors in tensor worker transport

No comparison

Serialize retained CPU batches without exporting one shared-storage descriptor per tensor.

General guidance. A bounded work queue does not bound the file descriptors retained by PyTorch shared tensor storage when the parent keeps the full source dataset. Copying each CPU batch into ordinary serialized bytes before queue submission avoids this storage-descriptor lifetime. Test with a low open-file limit and more distinct retained tensors than that limit. Check exact input tensors and output tokens. ARC evaluation failed after retaining 147 complete batches; the byte-transfer path passed the low-limit CPU regression. This is a reliability fix; no matched throughput gain is claimed.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Cache-aware expert routing

No comparison

Prefer resident experts inside a bounded router-rank window.

General guidance. Cache-aware routing can reduce storage misses by substituting a resident expert for a higher-ranked nonresident expert. It changes model execution and can increase other work even when hit rate rises. Record routing agreement, substitutions, output agreement, task quality, expert work, and end-to-end throughput.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

CPU memory bandwidth

No comparison

Use balanced memory channels and avoid bandwidth bottlenecks.

General guidance. CPU inference and host-resident experts can be limited by memory bandwidth. Record DIMM channel population and memory clock. Measure local and remote NUMA bandwidth before comparing hosts or placement policies. The R720 streams 38.4 GB/s from one socket and 73.2 GB/s from both with local memory, or 70.4 GB/s interleaved; its DDR3 RDIMMs are rated 1600 MT/s but run at 1333 MT/s. Compare a kernel with a plain read of the same memory before you write a faster kernel: a kernel already near the read rate cannot gain.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Direct storage I/O

No comparison

Bypass buffered page-cache I/O for streamed model tensors.

General guidance. Direct I/O can avoid page-cache copies and readahead limits on fast local storage, but the result is filesystem and device dependent. Compare it with buffered I/O on the actual model volume and keep it disabled when the repeated difference is within noise.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Fast RPC tensor upload

No comparison

Upload model weights to llama.cpp RPC workers without byte-wise hashing or staging copies.

General guidance. Upstream llama.cpp RPC hashes every tensor larger than 10 MB with byte-wise FNV-1a before it uploads it, so that a worker with a local cache (rpc-server -c) can skip it; it then copies the tensor into a new zero-filled buffer, and the worker receives it into another new zero-filled buffer. All of it runs on one thread per side, so a model load reaches the RPC workers at a few hundred MB/s however fast the link is: the R720 measured the same rate over loopback as over its 40 GbE link. Hashing eight bytes per step, sending the tensor without the staging copy, and reusing the worker's receive buffer keep the wire format and the computation. The hash values change, so a patched client and an unpatched cache miss each other and send the data in full. What remains is serial: one tensor at a time, a pageable host-to-device copy on the worker, and two kernel copies per byte; pinned, double-buffered receives would be the next step.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

FP16 orthogonal momentum products

No comparison

Use half precision for matrix products in orthogonalized momentum updates.

General guidance. Keep gradient normalization and scale calculations in FP32, then use FP16 tensor-core products for Newton-Schulz iterations on older GPUs. Check zero and small gradients, finite updates, and long-run skipped updates. This is a numerical port from upstream BF16, not proof of equal final quality. The corrected grouped-attention mdlARC probes measured optimizer time at about 11 to 13 percent of an update, so optimizing only this part cannot yield a tenfold total training speedup.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

GPUDirect RDMA

No comparison

Let an RDMA adapter access GPU memory without host staging.

General guidance. GPUDirect RDMA can remove host-memory copies from network transfers, but it does not guarantee higher end-to-end throughput. PCIe topology, GPU and adapter generations, DMA read behavior, message size, and collective direction can make direct GPU-memory reads slower than host staging even when direct GPU-memory writes are faster. Require an NCCL GDRDMA trace marker, validate CUDA-buffer verbs registration, and test NCCL_NET_GDR_READ=0 and 1 independently. Compare a matched message-size sweep before enabling either mode for a workload.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Group attention padding by sequence length

No comparison

Limit padding inside attention while preserving the training batch.

General guidance. Group similar lengths for attention, then restore token order before the next layer. Keep projections and feed-forward work packed, and reuse one layout across layers. Check dropout gradients across group boundaries. The mdlARC 40-update probes, one run per condition, reduced mean update time by 35 percent for RoPE and 39 percent for PoPE versus full-batch padded SDPA. The layout changes dropout random draws while preserving their rate and distribution; timing probes alone do not establish equal final quality.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Independent model replicas

No comparison

Run one model server per accelerator and balance complete requests.

General guidance. Replicas avoid tensor collectives and provide fault separation. They usually improve aggregate throughput when one model copy fits on each accelerator. Compare aggregate throughput and tail latency with a single model that uses the same devices.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Keep the model loaded between requests

No comparison

Reuse loaded weights and initialized runtime state for successive requests.

General guidance. A short inference can cost much less than importing the runtime, constructing the model, loading weights, and initializing kernels. Measure process-to-first-response separately from requests in the same loaded process. A persistent worker or streaming CLI can amortize startup, but needs an explicit unload or stop path. Residency is not a prompt cache: each classifier input still gets a complete forward pass. The Laya screen illustrates the difference; its single samples do not isolate each startup cost or establish a precise steady-state latency distribution.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Looped shared-weight transformer blocks

No comparison

Reuse a few transformer blocks several times instead of stacking distinct blocks.

General guidance. A looped model applies a small set of blocks in turn for the same number of block steps, so per-token arithmetic stays the same while stored weights shrink. On mdlARC at 90 epochs, 2 blocks x 4 passes (26% of the parameters) scored 0.55 points below 8 distinct blocks on held-out ARC-1 tasks across two seeds (95% interval -2.8 to +1.7), and 4 blocks x 2 passes tied with one seed; both trained 9-11% faster with 9-16% less GPU energy. It did not raise accuracy. The saving is weights, not arithmetic: every token still passes through the same number of block steps. Plain looping does not extend at test time: a model trained with 4 passes lost answers with 3 or 6 passes. Using looping for adaptive computation requires training with varying pass counts or a readout after each pass.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Multi-rail transport

No comparison

Use more than one matched network port for one distributed workload.

General guidance. Multi-rail transport can add bandwidth when the workload, collective algorithm, host path, and HCA uplink can use concurrent ports. It can also add setup and small-message latency, and nominal link speeds do not prove aggregate throughput. Compare each port with a matched dual-port run. Pin control traffic, select ports explicitly, validate the intended transport, and require per-port counters. Test host staging and each GPUDirect direction because another rail cannot remove a GPU-memory or shared-PCIe bottleneck.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Output-token-weighted data parallel training

No comparison

Share one global batch and normalize distributed gradients by output tokens.

General guidance. Variable-size examples require weighting each rank by its output-token count before the DDP average. Verify full-batch gradients on unequal examples. Gradient transport, sequence imbalance, and replicated optimizer work can exceed the compute saved. One ARC 14-update screen over the direct 40 GbE socket path was 14.9 percent slower than one V100; this is not a general four-card scaling result. Dropout and reduction order change with partitioning; timing alone does not establish final accuracy. Same-host ARC screen: 281.3 ms/update versus 376.1 ms on one V100, 1.337x update rate and 1.496x allocated GPU time. One run per arm, 12 timed updates; final accuracy unmeasured.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Packed variable-length training

No comparison

Avoid padding arithmetic during training with mixed sequence lengths.

General guidance. Packed projections and feed-forward layers avoid padding work. Packed attention also needs correct sequence boundaries and random masks. Test gradients with the actual nonzero dropout rate: the private PyTorch 2.11 varlen CUTLASS path passed zero-dropout checks but used different masks in forward and backward when dropout was enabled. The corrected ARC adapter pads only attention and uses public SDPA. No speedup follows from invalid-gradient timing runs.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Pinned host transfer buffers

No comparison

Copy to and from page-locked host memory when a bulk transfer matters.

General guidance. A copy from ordinary pageable memory is first a host memcpy into a staging buffer the driver owns, so it runs at host memory speed and cannot reach the link. Page-locked memory lets the device pull the bytes directly. Measure both, because the gap can be larger than the link itself: on the R720 V100s a pageable host-to-device copy reached 2.0 to 3.4 GB/s while a pinned copy over the same x16 link reached 11.4 to 12.1. A pageable path is also sensitive to the NUMA policy, and a pinned path is not.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Polar-coordinate position encoding

No comparison

Represent query and key content as positive magnitudes with explicit positional phases.

General guidance. A real/imaginary implementation can double query/key attention width while preserving projection width. This adds arithmetic and cache memory. Compare exact task scores as well as elapsed time at a matched update budget; a loss reduction alone does not establish better task performance. The ARC study uses a documented 3D adaptation with learned per-head key phase offsets. Its one-seed comparison raised training time and memory use without a clear confirmation-score gain. This is a workload-specific result, not a general rejection of polar encodings.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Prefix caching

No comparison

Reuse saved prompt state for repeated input prefixes.

General guidance. A prefix cache removes repeated prefill work. Test it first for any agent or chat workload, before decode tuning. The measured lab agent profile in results/20260817T000739Z-agent-context-profile sends a median prompt of 114,419 tokens of which a median of 1,153 tokens are new; 97.4% of all input tokens repeat a prefix the server has already processed. Repeating that prefill costs about 39 times the total prefill work of a cached service, and no decode optimization in this catalog moves served latency by a comparable factor. Test exact hits, near misses, first requests, and concurrent slots. Record time to first token, cache state, cache memory, and miss overhead. Report the hit and miss cases as separate results; a single average hides the case a user feels. Size the cache from measured concurrency, not from one session: the same profile peaked at 14 concurrent sessions and held a time-weighted mean of 3.88, so the service must retain several prefixes of 100K to 250K tokens at the same time. That capacity competes with weights and with the active KV cache, so pair this work with kv-cache-quantization and continuous-batching. A prefix cache does not increase decode speed for unrelated prompts. Validate the exact serving profile. On four V100s with input embeddings placed on CUDA0, Qwen3.5 Q5 and Q6 both failed the cold-versus-restored token check on b10488, although an earlier three-V100 single sample on that revision matched. A newer runtime alone does not establish cache correctness. Retain the first failed pair instead of repeating it for hours. A small CPU state diagnostic on b10488 separated append, host restore, and checkpoint rollback. Host restore preserved the returned scores. Cold/append score drift disappeared when checkpoint-driven prompt batch boundaries were equal. This does not establish the cause of four-V100 mismatches. See qwen35/results/20260917-cache-state-diagnostic/README.md. The Qwen Q6 four-V100 32K failure was isolated to its prompt batch schedule. Direct rollback reproduced the prior failed tokens without host-cache eviction. With checkpoints disabled, append-only replay of those batches reproduced every returned score; equal cold/cached batch boundaries gave exact agreement. Short GPU host restore and rollback controls also matched. CUDA batch and expert token counts can select different kernels and activation representations; the first differing tensor was not instrumented. This does not clear other models or 128K, or make checkpoint disabling a serving fix. See qwen35/results/20260917-gpu-cache-state/README.md. New sequential screens use aligned-prefix-v1: an exact token prefix ending on a full batch, equal batch and microbatch sizes, and checkpoints disabled in both arms. Clear the active slot before prefix preparation; use a short disjoint request for host eviction, then require the expected restored position, exact output tokens, and exact returned scores. Keep one model load per depth. The Qwen Q6 32K process check passed one cold/cached pair, including all 150 tokens and returned scores, at qwen35/results/20260917-aligned-cache-32k/. This is an aligned restore/append control, not a claim that arbitrary edits or other batch shapes are invariant. Bonsai 2 27B PQ2_0 on one V100 and PrismML runtime 5d80cff0b also passed aligned-prefix-v1 at both 32K and 128K, including all 150 output tokens and returned top-ten scores in each pair. Each state has one provisional sample. Prompt timings are API processing proxies; host-cache transfer is separately visible in the log and must not be omitted from an end-to-end latency claim. Evidence: bonsai2/results/20260917T225032Z-bonsai2-pq2-v100/.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Product-of-experts candidate selection

No comparison

Rank sampled answers by their product of probabilities under several augmented views.

General guidance. When a model generates many candidate answers under input augmentations, counting how often each answer appears treats every view as an independent vote. Many views can agree on a plausible wrong answer. Instead, re-encode each leading candidate under a fixed set of augmented views and sum its teacher-forced log-probabilities. This penalizes a candidate that any view finds unlikely. It needs one forward pass per candidate and view, not new generation, so it is cheap relative to sampling. It cannot recover answers that were never generated. On mdlARC (ARC-1) the product beat vote counting, a vote prior, and the minimum view; Franzen et al. report the product also beat mean, max, and min probability.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Two-stage microbatch pipeline training

No comparison

Split layers across two GPUs and overlap microbatch forward and backward work.

General guidance. Normalize microbatch losses by total output tokens, accumulate the full original batch, then unscale, clip and update once. Use separate stage launch threads and transfer streams. Check gradients and optimizer updates against serial staging at the actual dropout rate. Pipeline bubbles, small kernel batches, padding and transfers can erase compute gains; measure a matched control and count GPU time as well as elapsed time. Same-host ARC screen: 540.4 ms/update versus 376.1 ms on one V100, 0.696x update rate and 2.874x allocated GPU time. One run per arm, 12 timed updates; final accuracy unmeasured.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline