Configuration File Reference

Field-by-field reference for the two TOML files a simulation takes: a model config (configs/<name>.toml) and a workload (workloads/<name>.toml). Unknown fields anywhere in either file are rejected.

# configs/<name>.toml
model = "deepseek-v4-flash"    # catalog preset, or an inline [model] table

[scheduler]                    # engine args shared by every hardware entry
[speculative]                  # optional: speculative decoding, shared default
[router]                       # optional: how requests spread over replicas
[decode_router]                # optional: disagg decode pool (defaults to [router])
[memory]                       # optional: KV tiers beyond HBM, from the hardware's stores
[prefill]                      # optional: a prefill pool; the entry then decodes (disaggregated)
[fault]                        # optional: serve-mode static fault injection

[hardware.b200]                # one entry per hardware the model runs on
tp = 4
replicas = 4                   # identical workers behind the router
[hardware.gh200]
tp = 4
scheduler = { max_num_batched_tokens = 4096 }

# workloads/<name>.toml — the [workload] table at top level
arrival_pattern = "closed_loop"
num_concurrent_users = 256
num_requests = 2000
seed = 7
input_len_dist = { type = "lognormal", mean = 7.0, std_dev = 0.5 }
output_len_dist = { type = "lognormal", mean = 6.5, std_dev = 0.8 }

inference-lab --config <model config> --hardware <entry> --workload <workload> runs one; --hardware may be omitted when the file has one entry. In Rust, ModelConfig::from_file(..).deployment(Some("b200")) gives a Deployment (model on hardware) and .with_workload(WorkloadConfig::from_file(..)) a Config. The WASM API takes that resolved Config as JSON: hardware, parallel, model, scheduler, workload, optional replicas, router, decode_router, memory, prefill, speculative.

Catalog presets

Hardware and model presets ship inside the crate (catalog/hardware/*.toml, catalog/models/*.toml, embedded at build time). A config names one with model = "<name>" and [hardware.<name>]; inference_lab::catalog:: {hardware_names, model_names, hardware, model} list and load them from Rust. Presets carry only the physical hardware spec and the model architecture; everything about a deployment (TP, memory utilisation, batch limits) is in the config. To tweak a preset, copy its table inline.

[hardware.<name>]

One entry per hardware the model is deployed on. The entry name is the hardware preset unless spec says otherwise.

Collectives are priced on the hardware’s [fabric] (below) and added serially to the step; an entry with tp > 1 or ep > 1 on hardware without one is rejected. Per layer: attention → one all-reduce over tp (none under dp_attention); dense FFN, or MoE without dp_attention (any ep) → one all-reduce; under dp_attention, dense FFN or MoE with ep = 1 → an all-gather + reduce-scatter, MoE with ep > 1 → dispatch and combine all-to-alls over ep, each rank moving its tokens / ep share of experts_per_tok hidden vectors. Expert reads and FLOPs are taken as balanced across ranks; there is no overlap of collectives with compute.

Hardware spec

Per-GPU physical spec: a catalog preset (catalog/hardware/<name>.toml) or an inline spec table.

[fabric]

Cost per collective, added serially to the step (no overlap): a TP all-reduce of V bytes over g ≤ gpus_per_node ranks is latency + f·V / bandwidth with f = 2(g−1)/g (ring) or 1 (in-network reduction); over more ranks it is reduce-scatter and all-gather inside each node around an all-reduce of the V/k shard across the n nodes on the NIC. An EP all-to-all moves each rank’s (g−1)/g share at the scale-up rate, or, across nodes, its in-node and cross-node shares concurrently on their own links. dp_attention skips the per-layer all-reduce.

[memory] on the hardware

The stores a node offers for KV beyond HBM and the links that reach them, templated per GPU or per node: a graph. Presets ship one (host DRAM and a node NVMe pool behind each GPU’s PCIe port on the HGX parts; Grace LPDDR5X behind NVLink-C2C on gh200; NVLink ports into the node switch and one NIC per GPU to the network on all); an inline hardware table can declare its own.

Instances: a tp-GPU worker pools its GPUs’ per-GPU stores, junctions and ports (capacity and bandwidth × tp); gpus_per_node / tp workers share a node’s per-node instances; every worker points at the same selected cluster store. A cluster access runs GPU → its NIC → network → store (and back), so the NIC remains the per-worker cap even when the store is striped. A transfer takes the shortest hop path between its ends and runs at its max-min fair share on every edge of it: the most contended edge fixes its transfers’ rate first, the residual is shared among the rest. Tier promotions run store → GPU; peer_hbm promotions run from the selected sibling GPU through the node switch to the destination GPU; hand-offs run GPU → network → GPU (see kv_link_bw).

Shipped presets, at datasheet figures: b200 (192 GB / 8 TB/s), b300 (288 GB / 8 TB/s), gh200 (96 GB / 4 TB/s), h100 (80 GB / 3.35 TB/s); each carries its node’s fabric (8-GPU NVSwitch + CX-7/CX-8 for the HGX boxes, 4-GPU NVLink + Slingshot for GH200).

[model]

A model is described by its per-token weight streams and its token-mixing layer classes; there are no named architectures. Any transformer the simulator serves is a composition of the pieces below.

[[weights]] — a per-token GEMM stream at one precision

A dense fp8 model is one stream; DeepSeek-V4 is an fp4 expert stream with routing plus an fp8 non-expert stream; gpt-oss is fp4 experts + bf16 rest.

[[layers]] — a class of identical token-mixing layers

kind = "attention" — GQA / MHA with a growing KV cache:

kind = "mla" — multi-head latent attention, optionally sparse:

Kimi-K2 is one mla class (full context); DeepSeek-V4 is three (window 128 only; window + top-k of the ÷4 history with an indexer; window + the whole ÷128 history); GLM-5 is top-2048 over the uncompressed history.

kind = "linear" — linear attention / SSM (GatedDeltaNet, Mamba, KDA):

Shipped model presets: catalog/models/ (inference_lab::catalog::model_names()), each with the derivation of its numbers from the HF config in its header.

[scheduler]

Workload file

The [workload] table, at top level of workloads/<name>.toml.

[memory]

Optional; no tiering when absent. Which of the hardware’s stores hold KV evicted from HBM, closest first, and how much of each. A worker (a tp-GPU replica) reaches each tier over its own link — its GPUs’ ports pooled, so a tp = 2 worker on gh200 promotes at 2 × 450 GB/s from 2 × 120 GB of Grace memory. A per = "node" store is shared by the workers on that node (gpus_per_node / tp of them): what one demotes, its neighbours can promote. A worker wider than a node pools the node stores it spans.

[memory]
tiers = ["peer_hbm", "host_dram", "nvme"] # peer_hbm is declared by b200
preset = "reactive"                            # reactive | oracle (optional bundle)
source = { policy = "promote" }                # promote | min_time
hbm_eviction = { policy = "lru" }              # lru | outlook
write = { policy = "write_back" }              # write_back | write_through | selective | live
eviction = { policy = "fifo" }                 # fifo | lru | ttl | outlook
prefetch = { policy = "none" }                 # none | outlook
backup = "on_evict"                            # on_evict | on_land
hit_refresh = "first_tier"                     # first_tier | none
promote_fill = "through"                       # through | buffer | direct
storage_prefetch = { policy = "wait_complete" } # wait_complete | best_effort | timeout
load_overlap = "layerwise"                     # layerwise | none
hbm_evict_backed_first = false
[memory.capacity]
host_dram = 1.0e12          # bytes per instance given to KV

The outlook, live and min_time/prefetch policies act on a session step’s outlook: on a session workload, when a step completes the simulator knows its successor’s arrival (completion plus the recorded gap) and how much of the context it re-enters with, and marks those blocks — in HBM and in the tiers — with that time. Other workloads announce nothing, so under live nothing is written and outlook eviction reduces to LRU. The gap between the reactive and oracle presets on the same replay is the value of knowing the re-entry.

Under promote_fill = "through", tiers are inclusive: a promoted block keeps every tier copy it staged through (KV is immutable), so its next eviction from HBM is a free drop and only blocks no tier holds ever cost a write. buffer keeps only the backing source; direct changes no store residency. Writes are transfers GPU → store on the same graph as promotions (full duplex: they share a port with fetches only in the reverse direction, but do share an NVMe pool’s drives); a block is resident once its write lands and a promotion of a block still arriving waits for it (the write-before-reuse race). A full store evicts its victim into the next tier as a store → store transfer, or drops it; a block dropped without ever having been promoted counts as dead bytes. A prompt whose blocks sit below the closest store is staged upward one store at a time, sharing every traversed edge with whatever else is in flight. These legs hold no HBM. The final closest-store-to-GPU leg reserves the request’s landing blocks; direct mode instead reserves them for its one source-to-GPU transfer.

A staged leg is admission-controlled against its destination store, the way HiCache bounds a storage prefetch by host-pool free space: it starts only if the store can take its blocks by evicting nothing that is pinned, and on start it pins its destination copy — in flight and after landing — until the request’s final HBM load consumes it (or the stage is cancelled, times out, or is abandoned). A request whose stage does not fit yet holds at the head of its worker’s queue until a stage completion or consumption frees pinned room; one whose external prefix exceeds the destination store outright abandons the prefetch and recomputes the external suffix. Under storage_prefetch = timeout, the deadline starts when the stage starts, not while it waits for room.

pin is set on each hardware store entry. Staged reads always protect their store source until that leg drains. On a direct read, pin = true prevents the source from being recycled; with pin = false, completion lands only the prefix still present, releases the unused HBM reservation, and recomputes the suffix. Peer HBM defaults to pinned; normal stores preserve the unpinned direct-read default.

The summary’s memory section reports, per store name, blocks held, bytes written / read / dead, evictions and expiries; per link name, bytes moved and utilisation; and totals of bytes written, bytes promoted, peer-HBM bytes promoted, pin stalls, partial landings, and promotions that waited on a write. The prefix_cache section counts lookups recomputed instead of fetched (min_time) and prefetches started (prefetch = outlook), with their tokens.

[prefill]

Optional. A disaggregated topology: this block is the prefill pool, and the hardware entry — its hardware, tp/ep/dp_attention, replicas and [memory] — becomes the decode pool. Arrivals enter the prefill pool through [router], prefill there against that pool’s HBM and [prefill] memory tiers, and hand their KV to a decoder chosen by [decode_router] over the network: the prefill worker’s nic link, the core, the decode worker’s nic. The first token rides with the hand-off. KV moves one way: a decoder’s HBM shortens later hand-offs of the same prefix, but nothing flows back to the prefill side, so a re-entry whose prefix exists only in decode HBM is recomputed by prefill. A per = "cluster" store named in both pools’ tiers is the one shared tier.

[prefill]
replicas = 2                       # prefill workers
parallel = { tp = 8, dp_attention = true }
memory = { tiers = ["host_dram", "nvme"], preset = "reactive", backup = "on_land" }
kv_link_bw = 4e11                  # optional core cap, bytes/s

backup, hit_refresh and promote_fill are graph-wide and taken from the pools that have tiers. [scheduler] (including balance_set) applies to both pools.

[router] and [decode_router]

Optional; round_robin when absent. [router] picks the replica each arriving request enters (replicas on the hardware entry; the prefill pool on a disaggregated topology). [decode_router] picks the decode worker each hand-off goes to on a disaggregated topology, and defaults to [router]. The KV-reading policies look up each replica’s state for the prompt on every decision — an estimate from the replica’s block index, as a KV-aware front end sees it, not the scheduler’s admission-time lookup.

• round_robin cycles through the replicas.
• least_loaded picks the fewest requests in system (running + waiting), ties by queued prefill tokens, then index.
• prefix_affinity picks the replica holding the longest cached prefix of the prompt (any tier); with none anywhere it falls back to least_loaded.
• kv_aware minimises (prompt − cached prefix) + load_weight × queued prefill tokens: the prefill work the request adds plus the prefill work already ahead of it, in tokens. A prefill-side policy: on a decode pool the load term is always zero.
• kv_aware_decode minimises (context − prompt prefix resident in the decoder's HBM) + load_weight × running sequences: the KV the hand-off must move plus the decode batch it joins, in tokens. Decoders whose free KV cannot hold the incoming context are passed over while any can.

The summary’s router section (and decode_router on a disaggregated topology) reports requests per replica and, for the KV-reading policies, how many decisions had a cached prefix on some replica, how many went to a holder, and how many went away from the longest holder. handoff reports transfers, bytes moved, and bytes skipped because the chosen decoder already held the prefix. For session workloads, reusable_kv reports the joint prefill/decode residency of reusable KV, derived hit, recompute and transfer fractions, a parent-prefill, parent-decode, and inherited-context split of prefiller misses, and per-decode-rank detail. Under same-rank session affinity, that split separates eviction-driven recomputation from decoder-output KV that was never written back. hbm reports capacity, resident prefix bytes, active/reserved bytes, and actual HBM eviction bytes per worker. sessions reports completed and deadline-censored sessions plus turns per started session; simulation records the arrival deadline snapshot and post-deadline drain; work separates logical prompt tokens from positions actually computed by prefill.

[speculative]

Optional. Decode steps then verify 1 + draft positions and advance by 1 + accepted.