Two vLLM servers share one 128 GB unified pool with dgx-embedd, so most of these flags are an allocation contract rather than benchmark chasing: deterministic memory partitions, a prefix cache that makes turn N+1 nearly free, FP8 KV that buys the 262k context, and kernels autotuned for GB10.
- --served-model-name ornith
- The alias is the contract, not the checkpoint path. LiteLLM upstreams, agent roles and scripts keep pointing at `ornith` when the weights underneath are swapped.
- --host 0.0.0.0 --port 8009
- All-interface binding is what lets the LiteLLM container reach the engine across the container bridge; the fixed port keeps the proxy upstream static.
- --moe-backend b12x
- Blackwell-tuned MoE kernel. For an A3B model the per-token cost is expert dispatch plus dozens of small grouped GEMMs; b12x fuses them instead of launching one GEMM per expert — the single largest throughput lever here.
- --gpu-memory-utilization 0.40
- A partition, not a tuning knob: ~48 GiB of the 128 GB pool for the primary model while LFM3 (0.07) and dgx-embedd (5.5 GB) co-reside. Unified memory has no swap — oversubscribing it takes the host down, not just the process.
- --speculative-config "$SPECCONF"
- Draft-model speculation: a cheap proposer emits candidate tokens, the 35B target verifies them in a single forward pass. Because only ~3B parameters are active per token, verification is nearly free and acceptance is high — inter-token latency drops with zero quality change, since accepted tokens are verified rather than approximated.
- --kv-cache-memory 11542910976
- An explicit 10.75 GiB KV pool (≈234k FP8-cached tokens) instead of a percentage. Cache size becomes deterministic, so TTFT and concurrency do not drift when a co-tenant changes its own reservation.
- --enable-prefix-caching
- Every agent turn re-sends the same system prompt, tool schemas and file context. Cached prefix blocks skip prefill entirely, so turn N+1 pays only for the new tokens — the dominant TTFT win in a coding loop.
- --no-disable-cascade-attn
- Keeps cascading attention enabled: shared prefix blocks are reused across query chunks instead of being recomputed, which is what makes a 262k cache bandwidth-affordable.
- --max-num-seqs 10
- Concurrency cap. Ten sequences matches real agent concurrency (main session plus subagents); a wider batch multiplies KV pressure and trades every stream’s latency for aggregate tokens/s nobody is waiting on.
- --enable-flashinfer-autotune
- Autotunes the FlashInfer attention/prefill kernels for GB10’s exact SM count and caches the plan. Generic kernel heuristics leave double-digit percent on the table on a part this unusual.
- --max-model-len 262144
- The full native 256k window, so long agent sessions never truncate mid-file. Affordable only because KV is FP8 and the pool is reserved explicitly.
- --max-num-batched-tokens 8192
- Scheduler step budget. 8k tokens per step keeps decode interleaved with a large prompt’s prefill, so a 200k-token context does not freeze streaming for seconds.
- --trust-remote-code
- Required: the checkpoint ships custom modeling code, and vLLM refuses to instantiate the architecture without it.
- --enable-chunked-prefill
- Splits long prompts into the 8k chunks above so decode tokens keep flowing while a large prefill is in flight.
- --kv-cache-dtype fp8
- FP8 E4M3 K/V halves cache bytes per token to 48 KiB (2 × 48 layers × 4 KV heads × 128 head_dim × 1 B). Attention tolerates quantization far better than weights do, and this is the single change that makes 256k context fit at all.
- --load-format fastsafetensors
- Parallel multi-threaded tensor loading: restarts take tens of seconds instead of minutes — which matters because every flag above was found by restarting.
- --reasoning-parser qwen3
- Splits the thinking channel from the answer as structured fields, so the harness can run TTSR over reasoning without reasoning text leaking into the reply.
- --enable-auto-tool-choice --tool-call-parser qwen3_coder
- Turns the model’s native tool-call grammar into typed arguments instead of prose the harness has to re-parse — fewer failed calls, less retry prefill.