Sloptimized

Reusable techniques and measured applications

Optimization catalogue.

98% agent-driven model benchmarks and optimizations.

Optimizations

The catalog connects reusable optimization techniques to measured runs. The change column reports decode throughput against the named baseline configuration. A row can include several techniques, so the change does not isolate one technique's effect. Expand a row to see what changed, the limits of the result, and the retained evidence.

Standout improvements

Largest valid gains with recorded baseline output agreement.

+525.4%

Mistral Small 4 119B 2603

ms4d reached 75.1 decode tok/s versus ms4s.

+257.9%

GLM-5.2

col2g reached 0.6 decode tok/s versus colq1.

+218.4%

Qwen2.5 72B Instruct

q72d2 reached 8.3 decode tok/s versus q72s2.

+79.7%

GLM-5.2

r4fv0 reached 0.7 decode tok/s versus r4fs0.

Techniques and rows are ordered by improvement. Each section gives the generic guidance. Select a configuration row to show its application notes and retained evidence.

GPU layer offload

Best associated +951.0%

Place target-model tensors in accelerator memory.

General guidance. GPU memory usually gives more bandwidth than host memory. Increase the offloaded target layers until the model, KV cache, compute graphs, and service reserve fit. Use matched benchmarks because a placement change also changes the CPU and GPU work balance.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

GPU peer access

Best associated +951.0%

Use direct transfers between accelerators when the platform permits.

General guidance. Direct peer access can remove a host-memory copy from multi-GPU traffic. Topology, firmware, the operating system, and IOMMU policy can block or slow this path. Test bidirectional bandwidth and latency before and after peer access is enabled.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Same-host layer split

Best associated +951.0%

Place different model layers on local accelerators.

General guidance. A layer split combines accelerator memory and transfers activations at layer boundaries. It is often a useful first multi-accelerator mode when the model does not fit on one device. Tune the split for memory balance and keep the full target on one host when possible. On the R720 PCIe SYS path without peer access, Mistral Small 4 row split was 53.1% slower for decode than layer split. A layer split earns nothing for a model that fits on one card. Qwen3.8 27B Q4_K_M is 15.9 GiB against 32 GiB per V100, and single placement matched the 1:1 split at every point measured, including 45.604 against 46.441 tok/s at a 30,906-token prompt. The split also made the speculative arm unstable: three of five layer-split MTP arms spread between 11.7% and 26.8% across their samples, while no single-card arm exceeded 2%.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Architecture-specific build

Best associated +289.5%

Build the runtime for the CPU and accelerator instruction sets.

General guidance. A common compatibility build can omit useful CPU instructions or accelerator code. Pin the source revision, compiler, and target architectures. A compiler function target can specialize one hot path without raising the instruction-set floor of the complete engine. Do not run a newer instruction-set path on an older host.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Kernel-path tuning

Best associated +289.5%

Select and tune kernels for the exact model and batch shapes.

General guidance. Automatic selection can use a slow fallback on an older accelerator or an uncommon tensor shape. Compare the supported matrix-multiply paths and exact-shape kernels. Record build flags, memory use, prefill speed, decode speed, and output checks. An exact grouped-int4 CPU path can vectorize independent output rows while preserving the scalar reduction order within each row. On dual Ivy Bridge E5-2680 v2 CPUs, Colibri PR 1286 measured exact kernel speedups of 2.153989 times at 20 threads and 2.655422 times at 40 threads. A matched five-sample streaming GLM-5.2 comparison measured a 1.796942-times median decode gain and 41.644% lower median TTFT, with identical text and canonical-token hashes. The candidate throughput range was wide, so retain and report the full range.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Weight quantization

Best associated +289.5%

Use lower-precision model weights to reduce storage and memory use.

General guidance. A smaller weight format can let more tensors stay in fast memory. The gain depends on hardware support, dequantization kernels, tensor shapes, and batch size. Compare task quality as well as speed. Do not transfer a result between different quantized artifacts.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Asynchronous expert I/O

Best associated +257.9%

Overlap out-of-core expert reads with useful model computation.

General guidance. An out-of-core MoE runtime can issue missing-expert reads while resident experts execute. The gain depends on storage latency, worker count, CPU contention, cache residency, and how much useful matmul is available to hide the reads. Measure a long enough decode to reach the later expert working set; a short warm screen can substantially overstate steady throughput. Track felt wait separately from read-service time and retain enough warm samples for the buffered page cache to converge.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

MoE expert placement

Best associated +257.9%

Place routed-expert tensors independently from other model tensors.

General guidance. A sparse MoE model uses only some experts for each token. Keep high-value shared and non-expert tensors on the GPU, then tune how many expert blocks remain on the CPU. The best split depends on expert routing, memory bandwidth, context size, and free VRAM. A runtime with expert-level staging can extend the split across VRAM, host RAM, and storage. Record the hit rate, bytes fetched per token, and read-wait share because correct output alone does not show that the storage tier is fast enough for the workload. On multi-GPU hosts, confirm that the backend selected every intended device and release redundant host copies before using the result to size RAM. A remote accelerator can be the expert-capacity tier. Qwen3.5 122B-A10B Q5_K_M replaced a 25,554.57 MiB CUDA_Host model buffer with a 16-layer remote V100 share, and every retained three-card sample exceeded every fitted two-card CPU-offload sample.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

NUMA memory policy

Best associated +257.9%

Place CPU memory near the device that consumes it.

General guidance. CPU tensors, draft work, and transfers can use different NUMA paths. Compare local allocation with interleave for the selected workload. Record the memory policy, CPU set, device node, and memory bandwidth. A backend NUMA flag such as llama.cpp --numa distribute places threads. It does not set the allocation policy of a private no-mmap weight buffer. Wrap the process in numactl --interleave=all to spread that buffer across the memory controllers. On the R720 this raised Laguna S decode about 30%. Interleave needs comparable free pages on every node; an unbalanced node makes results vary between runs. For a mmap-only backend, drop the page cache once and load under numactl; the file pages fault interleaved and the placement persists warm. This raised Qwen 72B prima decode 47.8%. The gain is not universal: a CPU-only Gemma 4 26B-A4B profile measured within noise of its mapped control, so measure the policy for each workload instead of assuming the gain.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Release GPU capacity

Best associated +257.9%

Remove auxiliary work from the target accelerator.

General guidance. Other GPU services use memory and scheduling time that the target needs. Stop them or move them to another accelerator or host. This can release memory for weights, KV cache, compute graphs, and draft models. Record the service placement for each test.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Speculative decoding

Best associated +65.8%

Use a cheaper predictor to propose tokens for target verification.

General guidance. A draft model or an MTP head proposes tokens before the target verifies them. Net speed depends on acceptance, draft latency, verification cost, and synchronization. Record a matched target baseline, output agreement, accepted-length distribution, proposal policy, and draft placement. Correct draft metadata is required. Muse Glimmer 30B on one V100 showed that useful width is both target- and quantization-specific: width one improved the 17 GB artifact by 20.1 percent, while width two improved the dynamic artifact by 34.8 percent. Width 16 reduced speed for both, and wider proposals caused output divergence on the 17 GB artifact. Qwen3.8 27B refined the width rule on a dense target: width one gave 1.38x, width two 1.43x, widths three and four 1.31x, and width six 1.01x on one V100, while acceptance fell from 87.3% to 43.0%. A high acceptance rate at a wide setting does not indicate a gain. The same head gained more at depth, 1.64x at a 30,906-token prompt with 88.1% acceptance, and more with generated length, 1.36x at 128 tokens against 1.52x at 1,024. Measure a speculative head on a current backend build: the same arm measured 1.27x on llama.cpp b9190 and 1.43x on b10358.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Speculative policy tuning

Best associated +65.8%

Tune proposal depth, lookahead, and verification branching.

General guidance. More proposals help only while accepted tokens repay draft and verification cost. Sweep small depths and any parallel proposal count. Stop when acceptance length or end-to-end throughput falls. Use the same target, prompt, sampling, and output checks. A minimum draft probability is not a default. Gating Qwen3.8 27B at 0.60 with width four reached 1.26x against 1.31x ungated at the same width and 1.43x at width two, so the gate lost on this hardware even though it raised acceptance from 0.57 to 0.71.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Draft placement

Best associated +52.2%

Place speculative draft work away from the target bottleneck.

General guidance. A draft can run on the target device, another accelerator, a CPU, or a remote host. The best location depends on free memory, draft latency, target interference, and transfer delay. Token-level traffic can tolerate a slower network than tensor traffic.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Host page cache

Best associated +47.8%

Keep mapped model files in the operating-system page cache.

General guidance. Large system RAM can make model reloads much faster and can let processes share clean mapped pages. Separate cold and warm start tests. Use proportional set size and major page faults to distinguish shared cache from private resident memory.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Cross-host layer split

Best associated +46.8%

Place contiguous model layers on accelerators in different hosts.

General guidance. A cross-host layer split combines device memory while transferring activations only at layer boundaries. It is useful when the remote capacity replaces CPU weights or compute; it is not automatically useful when the model already fits locally. Tune the remote share to its compute and transport path instead of assigning equal shares. Laguna S over llama.cpp RPC TCP and a measured 39.395 Gb/s direct link improved from 32.1608 to 38.9215 tok/s with 21:21:7 layers, while an approximately equal three-card split fell to 26.3659. Automatic fit can replace a requested split; disable it for manual placement and verify both layer assignments and model-buffer sizes. Qwen3.5 122B-A10B Q5_K_M confirms the capacity case and shows that the useful share is model-dependent. Its two-card fit retained 25,554.57 MiB in CUDA_Host, while an equal 17:17:16 three-card split removed that host model buffer and reproduced medians of 28.0916 and 27.5660 tok/s. The two-card controls were variable, so no exact median speedup is claimed; the slowest three-card sample was still 1.603689 times the fastest control sample.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Batch-size tuning

Best associated +39.9%

Tune logical and physical batch sizes for the workload.

General guidance. Batch and microbatch sizes change memory use, prompt throughput, and decode scheduling. They can also select different backend kernels. Large values can help prompt processing. Smaller values can release memory for more GPU layers, cache, parallel slots, or a draft model. The direction differs by architecture: a dense partial offload preferred batch 128 to gain GPU layers, but a hybrid MoE prefill processes CPU-resident experts in microbatch-size chunks, and microbatch 1024 nearly doubled Laguna S 4K prefill. Decode was insensitive to the same change.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Output tensor placement

Best associated +39.9%

Keep frequently used input and output tensors in accelerator memory.

General guidance. A hybrid model can still move data for token embeddings or output projection on each step. Place these tensors on the accelerator when they fit. Measure transfer traffic, memory use, and decode speed with a matched control.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Runtime selection

Best associated +39.9%

Select a supported runtime for the hardware and workload.

General guidance. Runtime support, model format, quantization kernels, and serving features change the result. Pin a compatible version and isolate legacy stacks. Compare the same model quality and workload. A newer runtime or a listed quantizer does not guarantee hardware support.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

CPU affinity

Best associated +37.7%

Bind inference threads to a selected CPU set.

General guidance. CPU affinity can keep work on the intended sockets and memory paths. It is most useful when CPU tensors and GPU transfers share a NUMA system. Record the exact CPU mask and the NUMA policy.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Flash attention

Best associated +37.7%

Use the backend fused attention path.

General guidance. Flash attention can reduce attention memory traffic and temporary storage. Its benefit depends on the backend, GPU architecture, context length, KV-cache type, and batch shape. Confirm that the runtime uses a supported kernel. Validate output and memory use with a matched control. The smaller compute buffer is a second lever: in a VRAM-limited partial offload it can free space for a larger microbatch or another expert block, which can be worth more than the kernel speedup itself.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Private weight copy

Best associated +37.7%

Use private host-memory model weights instead of mapped files.

General guidance. A no-mmap load uses private host memory for CPU-resident weights. This can avoid mapped-file behavior during mixed CPU and GPU inference, but it increases resident host memory. Compare it with a matched mapped control. The private buffer follows the process NUMA memory policy. On a multi-socket host, pair it with numa-memory-policy interleave, or the allocation lands on one node and loses the other memory controllers.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

CPU thread tuning

Best associated +18.8%

Match inference threads to the useful physical CPU resources.

General guidance. Thread count controls memory pressure, synchronization, and NUMA traffic. Test physical-core counts and nearby values. Keep batch threads separate when the backend supports this control.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Distributed target inference

Best associated -0.9%

Split target-model work across more than one host.

General guidance. A layer or pipeline split can add memory and compute capacity, but network transfer and synchronization can remove the gain. Token messages need much less bandwidth than tensor or KV-cache traffic. Compare the split with a one-host target. Record each node role, transfer volume, and the physical interconnect.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

KV-cache quantization

Best associated -4.5%

Use lower-precision key and value cache entries.

General guidance. A smaller KV cache can increase context length or parallel slots. It is not always faster and can change output quality. Compare F16, a supported 8-bit format such as FP8 or Q8, and Q4 as separate series with fixed prompts and sampling. Some parallel modes or attention kernels require a specific cache type. On the V100 with flash attention, Q8_0 KV decoded about 8% slower than F16 for Laguna S; quantized cache pays dequantization work and only wins when the freed memory buys capacity the profile needs. An agent serving profile is the clearest case of such a need: the freed memory becomes retained prefix slots, and one avoided full prefill repays a large decode loss. Judge the trade with prefix-caching enabled and with the measured concurrency, not on a single stream.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Tensor parallelism

Best associated -14.9%

Split model tensors across accelerators for each compute step.

General guidance. Tensor parallelism can reduce latency and combine device memory. It also needs frequent collective communication. Test it only after the peer and collective paths work. Record the split, cache type, fused collective options, and communication time. Runtime and model shape support are prerequisites: llama.cpp 9190 could not load its experimental tensor split for Mistral Small 4. A heterogeneous cross-host split can execute correctly while losing most throughput when generic RPC TCP carries the per-layer reductions. A CUDA/NCCL MLP proof on the V100/P40 pair matched its local reference while every channel used RoCE GDRDMA, although the heterogeneous two-rank step was still slower than one V100 executing both shards. Verify the actual transport and compare against a local execution control. On a SYS-connected pair without peer access, tensor split is a loss. One Qwen3.8 27B process decoded 32.965 tok/s under layer split and 31.932 tok/s under tensor split, with tensor samples between 18.568 and 33.856 tok/s, and native MTP fell to 26.131 tok/s under tensor split against 43.257 under layer split. Published tensor-split gains from cards under one host bridge do not transfer to a socket-crossing pair.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Cache-aware expert routing

No comparison

Prefer resident experts inside a bounded router-rank window.

General guidance. Cache-aware routing can reduce storage misses by substituting a resident expert for a higher-ranked nonresident expert. It changes model execution and can increase other work even when hit rate rises. Record routing agreement, substitutions, output agreement, task quality, expert work, and end-to-end throughput.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Direct storage I/O

No comparison

Bypass buffered page-cache I/O for streamed model tensors.

General guidance. Direct I/O can avoid page-cache copies and readahead limits on fast local storage, but the result is filesystem and device dependent. Compare it with buffered I/O on the actual model volume and keep it disabled when the repeated difference is within noise.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

GPUDirect RDMA

No comparison

Let an RDMA adapter access GPU memory without host staging.

General guidance. GPUDirect RDMA can remove host-memory copies from network transfers, but it does not guarantee higher end-to-end throughput. PCIe topology, GPU and adapter generations, DMA read behavior, message size, and collective direction can make direct GPU-memory reads slower than host staging even when direct GPU-memory writes are faster. Require an NCCL GDRDMA trace marker, validate CUDA-buffer verbs registration, and test NCCL_NET_GDR_READ=0 and 1 independently. Compare a matched message-size sweep before enabling either mode for a workload.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Independent model replicas

No comparison

Run one model server per accelerator and balance complete requests.

General guidance. Replicas avoid tensor collectives and provide fault separation. They usually improve aggregate throughput when one model copy fits on each accelerator. Compare aggregate throughput and tail latency with a single model that uses the same devices.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Multi-rail transport

No comparison

Use more than one matched network port for one distributed workload.

General guidance. Multi-rail transport can add bandwidth when the workload, collective algorithm, host path, and HCA uplink can use concurrent ports. It can also add setup and small-message latency, and nominal link speeds do not prove aggregate throughput. Compare each port with a matched dual-port run. Pin control traffic, select ports explicitly, validate the intended transport, and require per-port counters. Test host staging and each GPUDirect direction because another rail cannot remove a GPU-memory or shared-PCIe bottleneck.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Pinned host transfer buffers

No comparison

Copy to and from page-locked host memory when a bulk transfer matters.

General guidance. A copy from ordinary pageable memory is first a host memcpy into a staging buffer the driver owns, so it runs at host memory speed and cannot reach the link. Page-locked memory lets the device pull the bytes directly. Measure both, because the gap can be larger than the link itself: on the R720 V100s a pageable host-to-device copy reached 2.0 to 3.4 GB/s while a pinned copy over the same x16 link reached 11.4 to 12.1. A pageable path is also sensitive to the NUMA policy, and a pinned path is not.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline

Prefix caching

No comparison

Reuse saved prompt state for repeated input prefixes.

General guidance. A prefix cache removes repeated prefill work. Test it first for any agent or chat workload, before decode tuning. The measured lab agent profile in results/20260817T000739Z-agent-context-profile sends a median prompt of 114,419 tokens of which a median of 1,153 tokens are new; 97.4% of all input tokens repeat a prefix the server has already processed. Repeating that prefill costs about 39 times the total prefill work of a cached service, and no decode optimization in this catalog moves served latency by a comparable factor. Test exact hits, near misses, first requests, and concurrent slots. Record time to first token, cache state, cache memory, and miss overhead. Report the hit and miss cases as separate results; a single average hides the case a user feels. Size the cache from measured concurrency, not from one session: the same profile peaked at 14 concurrent sessions and held a time-weighted mean of 3.88, so the service must retain several prefixes of 100K to 250K tokens at the same time. That capacity competes with weights and with the active KV cache, so pair this work with kv-cache-quantization and continuous-batching. A prefix cache does not increase decode speed for unrelated prompts.

StatusCfgModelHostHardwareDecode tok/sΔ baselineBaseline