Ariff Nazhan

KAI Scheduler vs HAMi

Sun, 18 Oct 2026 00:00:00 GMT

KAI Scheduler and HAMi both decide which GPU a pod gets, and both can put several pods on one card. They solve different problems. KAI Scheduler decides who gets GPUs and when: queues, quotas, fair share and gang scheduling. HAMi decides which card and how much of it: it slices cards, caps memory per container, and lets a pod name the exact cards it runs on.

In short, they work at different levels. KAI is a Kubernetes scheduler: it works at the workload level and decides which pod or job runs, when, and on which node. HAMi works at the GPU level: it decides which card a pod gets and how much of it, and HAMi-core enforces that share inside the container. HAMi does not replace the Kubernetes scheduler, it extends it.

Where each one sits. Blue is KAI, green is HAMi.

We run several workloads on the same GPU nodes, and we had trouble putting each of them on specific GPUs. We tried both. This post compares them. The versions are HAMi 2.9.0 and KAI Scheduler v0.18.6.

What each one is

HAMi:

It is the device plugin, plus a scheduler extender and a webhook. It hands out the cards itself.
A pod can name the exact cards it runs on, by UUID.
A pod asks for a share of each card's memory, and HAMi-core can enforce that share inside the container.
It has no queues, quotas or gang scheduling.

KAI Scheduler:

It is a scheduler only. The cards come from the NVIDIA device plugin or from DRA.
A pod cannot name its cards. The device plugin picks them.
Card sharing works through reservation pods that hold whole cards. Memory is not enforced by default.
It has queues with quotas, fair share, preemption and gang scheduling.
It is bigger: 7 controllers, 6 CRDs and 4 PriorityClasses, against HAMi's DaemonSet, Deployment and webhook.

How a pod that shares two cards gets them. The shaded step is where the physical card is chosen. Drawn from HAMi's architecture and KAI's GPU sharing docs.

What each one is good at

KAI Scheduler, after its scheduling deep dive. Blue is where KAI works.

KAI Scheduler (v0.18.6) is built around queues and gangs. A pod-grouper puts related pods into PodGroups that start together or not at all. The groups wait in a tree of queues, each with a guaranteed quota, a limit, an over-quota weight and a priority. Every scheduling cycle allocates first, then consolidates running pods to cut fragmentation, reclaims over-quota usage between queues, and preempts lower-priority work inside a queue, with fairness by DRF and optionally by usage over time. Priority and preemptibility are separate settings, and a minimum runtime protects a pod from early preemption. On top of that it places by binpack or spread, can be topology-aware, runs elastic jobs with a minimum and maximum size, shares GPUs with gpu-fraction or gpu-memory, schedules DRA ResourceClaims, works with cluster autoscalers, and records each decision as a Kubernetes Event.

HAMi, after its architecture diagram. Green is where HAMi works.

HAMi (v2.9.0) is built around the device. Its device plugin splits each card into slots, and a pod asks for N different cards and a share of each: memory in MiB or percent, and compute as a percent of the SMs. The pod can name its cards by UUID (nvidia.com/use-gpuuuid) or by model, or keep away from some. HAMi-core enforces those shares inside the container with no change to the application, and the container sees its cap as the card's memory; CUDA_DISABLE_CONTROL=true turns it off for one container. Placement is binpack or spread, for nodes and for cards, chosen per pod, and dynamic MIG is available per pod. It supports accelerators from several vendors, exports what each pod holds on each card, and needs no CRDs: a scheduler extender on kube-scheduler, a device plugin and a webhook.

Things to know before choosing

KAI cannot pin a pod to given cards. The device plugin picks them, so the only way to land on specific cards is to have them be the only free ones when the pod binds, and the placement can change after a restart. Its GPU sharing needs the nvidia.com/gpu.memory node label from GPU Feature Discovery; without it every bind of a gpu-fraction pod is rolled back. In v0.18.6 it also counts admin-access DRA claims as GPU usage, so with the GPU Operator's DRA driver a node looks fuller than it is. Uninstalling it through ArgoCD leaves its CRDs and the namespace of its reservation pods behind, and leftover reservation pods keep holding cards.

HAMi cannot run next to the NVIDIA device plugin, since both advertise nvidia.com/gpu. It counts slots, not cards: with deviceSplitCount: 10, an 8-GPU node shows nvidia.com/gpu: 80.

With either tool, engines that share a card should start one at a time. vLLM measures free GPU memory when it starts, and two engines measuring at once count each other's memory wrong.

Why we use HAMi

Our problem was placement: several workloads, each meant for specific cards, some sharing a card. HAMi covers that in one component.

Pinning. A pod lists the card UUIDs it may use. The spec is the placement, and it survives restarts.
Sharing by memory share. A pod asks for N cards and a share of each. HAMi will not place a pod past 100 % of a card.
Enforcement per container. vLLM sizes its own memory, so vLLM pods run with HAMi-core off and the share is bookkeeping. Small services run with HAMi-core on, so they cannot grow into a neighbour's memory.
Small footprint. A device plugin, a scheduler extender and a webhook, no CRDs.

A vLLM engine pinned to two cards that it shares with another engine:

metadata:
  annotations:
    nvidia.com/use-gpuuuid: "GPU-aaaa...,GPU-bbbb..."   # exactly these two cards
spec:
  runtimeClassName: nvidia
  containers:
    - name: vllm
      env:
        - name: CUDA_DISABLE_CONTROL      # HAMi-core off: vLLM manages its own memory
          value: "true"
      resources:
        limits:
          nvidia.com/gpu: "2"                   # two different cards
          nvidia.com/gpumem-percentage: "45"    # of each card

When KAI is the better choice

KAI is built for a shared training or batch cluster: many teams, jobs that come and go, quotas that must hold, gang scheduling for multi-node training, and preemption between priorities. If which card a pod lands on does not matter and the problem is who gets GPUs first, KAI solves that and HAMi does not.

If the problem is which card, and every card has a planned owner, HAMi does it directly. KAI gets there only through rollout order, with a reservation-pod layer and a GPU Feature Discovery dependency on top.

Proxmox with GPU passthrough

Sun, 11 Oct 2026 00:00:00 GMT

A k3s cluster with one control plane and two workers needs three machines. I have one PC: an i9-10900K, 32 GB of RAM and one RTX 3090. So the PC runs Proxmox VE as the hypervisor and the three nodes are virtual machines, one of them with GPU passthrough enabled.

This post covers the Proxmox side only: the three VMs, PCI passthrough of the RTX 3090 to one of them, and static DHCP addresses on the router. The k3s installation is a separate post.

The PC

Part	What it is
CPU	Intel i9-10900K, 10 cores, 20 threads
RAM	32 GB
Disk	2 TB, ZFS
GPU	NVIDIA RTX 3090, 24 GB
Hypervisor	Proxmox VE 9.2.2
Network	one bridge, `vmbr0`, on my home LAN. The host is `192.168.100.5`

Proxmox is managed from a web UI on port 8006. It shows two storages on this host: local holds ISO images, local-zfs holds the VM disks.

Installing Proxmox: keep host drivers off the GPU

For passthrough, no driver on the Proxmox host may hold the GPU. A driver that has bound the card must release it before a VM can take it. The simpler state is that no host driver ever binds it.

The kernel in Proxmox VE 9.2 has three drivers that match an RTX 3090:

Module	What it is
`nouveau`	the older open-source NVIDIA driver
`nova_core` (with `nova_drm`)	the new open-source NVIDIA driver in recent kernels
`nvidiafb`	an old framebuffer driver

Blocking them starts at the installer. At the installer's boot menu, e opens the selected entry for editing. I added two parameters to the end of the line that starts with linux, then booted it with Ctrl+X:

nomodeset modprobe.blacklist=nova_core,nova_drm,nova,nouveau

Parameter	Effect
`nomodeset`	the kernel loads no mode-setting graphics driver; the console stays on the plain framebuffer the firmware set up
`modprobe.blacklist=...`	the listed modules are not loaded automatically, from the start of boot

The installed system has the same settings in two places. The kernel command line, /etc/kernel/cmdline:

root=ZFS=rpool/ROOT/pve-1 boot=zfs nomodeset modprobe.blacklist=nova_core,nova_drm,nova,nouveau

And a modprobe file, /etc/modprobe.d/pve-blacklist.conf:

blacklist nova_core
blacklist nova_drm
blacklist nova
blacklist nouveau
blacklist nvidiafb

The result can be read on the host while the GPU VM runs:

lspci -nnk -s 02:00

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau, nova_core
02:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

"Kernel modules" lists the drivers that could handle the device. "Kernel driver in use" is the one that has it: vfio-pci, the stub driver Proxmox uses to hand a device to a VM. lsmod on the host shows no nouveau, nova or nvidia module loaded.

The plan: three VMs

VM	ID	vCPU	RAM	Disk	What it becomes
`vm-1`	100	4	4 GB	60 GB	k3s control plane
`vm-2`	101	4	4 GB	60 GB	k3s worker
`vm-gpu`	102	8	16 GB	200 GB	k3s worker with the RTX 3090

The GPU VM gets most of the resources because GPU workloads also need CPU, memory and disk for the model files. The other two only have to run Kubernetes itself and small services.

All three run Ubuntu Server 24.04.

Step 1: the Ubuntu ISO

Proxmox can download an ISO by itself, so nothing has to go through my laptop. In the UI: local storage, ISO Images, Download from URL. Paste the URL, press Query URL, and Proxmox fills in the file name and the size.

The download dialog after "Query URL", before the checksum is filled in.

Before pressing Download I set Hash algorithm to SHA-256 and pasted the checksum from Ubuntu's SHA256SUMS file. Proxmox then verifies the 3.80 GiB file itself and refuses it if it does not match.

Step 2: two plain VMs

vm-1 and vm-2 are the Create VM wizard with its defaults, except three values: the ISO from step 1, a 60 GB disk on local-zfs, and 4 cores with 4096 MiB of memory.

The wizard's last tab for vm-1. vm-2 is the same with ID 101.

The defaults mean a SeaBIOS firmware and the i440fx machine type. That is fine for a VM with no physical device attached.

Step 3: the GPU VM

What passthrough needs

PCI passthrough hands one physical device to one VM. The VM's operating system loads the normal driver and uses the card directly. The host and the other VMs cannot use it while that VM runs.

It has two requirements on the host. The first is the one above: no host driver holds the card. The second is the IOMMU (Intel calls it VT-d), a CPU and motherboard feature that limits a device to the memory of the VM it is given to. The IOMMU puts devices into groups, and a group can only be passed through as a whole. So the question for any card is: what else is in its group?

On this host the IOMMU was already active when I looked. Proxmox listed 16 groups, and the answer was the good one:

The device list in the "Add: PCI Device" dialog. The RTX 3090 (0000:02:00.0) and its HDMI audio function (0000:02:00.1) are the two devices in IOMMU group 13. Nothing else comes with them.

The VM settings

Five settings differ from the two plain VMs:

Setting	Value	Why
Machine	`q35`	the default `i440fx` has no PCI Express; `q35` does
BIOS	OVMF (UEFI), Pre-Enroll keys off	with the keys enrolled, Secure Boot is on and the NVIDIA kernel module has to be signed before it loads
CPU type	`host`	the guest sees the real CPU and its flags instead of a generic model
Memory	16384 MiB, Ballooning Device off	all memory of a VM with a passed-through device is pinned, so the balloon cannot give any of it back
Disk	200 GB	room for container images and models

The first two are on the wizard's System tab. Choosing OVMF adds an EFI disk, which needs a storage:

System tab: q35, OVMF, the EFI disk on local-zfs, and "Pre-Enroll keys" unticked.

The last tab for vm-gpu. "Start after created" stays unticked, because the GPU is not attached yet.

Attaching the card

The wizard has no step for PCI devices. After the VM exists: Hardware, Add, PCI Device, Raw Device, pick 0000:02:00.0, then tick two boxes:

All Functions passes 02:00.0 and 02:00.1 together, the card and its audio function.
PCI-Express presents it to the guest as a PCI Express device. This is the option that needs q35.

The finished hardware list. The last line, hostpci0: 0000:02:00,pcie=1, is the GPU. The network card's MAC address is covered.

The GPU VM takes the monitor

Proxmox lists no other display adapter on this PC, so the RTX 3090 is also what drives the host's monitor. The moment vm-gpu starts, the card belongs to the VM and the monitor goes dark. The web UI and SSH keep working, because they do not need the screen.

That matters after a host reboot. If vm-gpu started by itself at boot, the host would have no screen, and I could not get to it from the monitor.

So the rule for this host: vm-gpu never has "Start at boot" on. The setting is on the VM's Options tab and is off by default. I checked it after creating the VM, and it is the first thing to check again after changing that VM. After a reboot the screen stays with the host until I start the GPU VM by hand.

The second thing to know is memory. With all three VMs running, the host showed 27 of 31 GiB in use. The VMs add up to 24 GB, and the GPU VM's 16 GB are pinned. There is no room for a fourth VM.

Step 4: install Ubuntu

Each VM boots the ISO, and the installer runs in the VM's Console tab. I took the defaults, with two things to note:

Network: leave it on DHCP. The installer shows the address the router handed out, for example 192.168.100.54/24 on vm-2. Write the three addresses down, they are needed in step 5.
SSH: tick "Install OpenSSH server", or the console stays the only way in.

One default I did not expect: Ubuntu's installer gave the root filesystem only half of each disk, 29 GB of the 60 GB and 97 GB of the 200 GB. The rest is free space in the LVM volume group. I have not extended it yet.

Step 5: fixed addresses on the router

The VMs get their addresses from my router by DHCP, and a DHCP address is a lease: it can change. That will matter later, because the k3s workers find the control plane by its address. If vm-1 comes back with a different one, the workers lose the control plane.

There are two ways to pin an address:

Way	Where it is set	What I chose
Static IP	inside each VM	no
Static DHCP (a reservation)	on the router, per MAC address	yes

With a reservation the router always hands the same address to the same network card. Nothing is configured inside the VMs, and the addresses are all in one place.

The router identifies a card by its MAC address. Proxmox shows it on the VM's Hardware tab, in the Network Device (net0) line, as virtio=BC:24:11:…. Proxmox generates these, they all start with BC:24:11, and a VM keeps its MAC as long as the network device is not removed and added again.

My internet is TIME, and on its router the page is called DHCP Static IP Configuration: New, type the MAC address and the IP address, Apply. One line per machine:

Machine	Address
Proxmox host	`192.168.100.5`
`vm-1`	`192.168.100.53`
`vm-2`	`192.168.100.54`
`vm-gpu`	`192.168.100.55`

The router page with the four reservations. MAC addresses are covered.

I did not pick .53 to .55. They are the addresses DHCP had already given the three VMs, and I reserved exactly those. A machine that holds a lease keeps it until it renews, so reserving the address it already has changes nothing on the VM. Reserving a different one would only take effect at the next renewal or reboot.

The router's DHCP range is 192.168.100.2 to 192.168.100.254. The four reserved addresses are inside that range, and this router accepts that.

Check: the card inside the VM

Passthrough is only proven from inside the guest. In vm-gpu:

lspci -nnk | grep -iA3 nvidia

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3090] [10de:2204] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GA102 [GeForce RTX 3090] [1462:3881]
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia

The card is at 01:00.0 in the guest and 02:00.0 on the host. The guest has its own PCI layout, the two numbers do not have to match.

"Kernel driver in use: nvidia" is there because I had installed Ubuntu's nvidia-driver-580-server package and rebooted. With the driver loaded, nvidia-smi reports the whole card:

nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv,noheader

NVIDIA GeForce RTX 3090, 24576 MiB, 580.178.04

What it looks like at the end

	`vm-1`	`vm-2`	`vm-gpu`
Address	`192.168.100.53`	`192.168.100.54`	`192.168.100.55`
vCPU / RAM / disk	4 / 4 GB / 60 GB	4 / 4 GB / 60 GB	8 / 16 GB / 200 GB
Machine and firmware	`i440fx`, SeaBIOS	`i440fx`, SeaBIOS	`q35`, OVMF
GPU	none	none	RTX 3090, all 24 GB
Start at boot	off	off	off, and it stays off

Three machines with fixed addresses, one of them with a GPU that works. The next step is k3s: vm-1 as the control plane, vm-2 and vm-gpu as workers.

RoCE vs NVLink for KV transfer

Fri, 02 Oct 2026 10:00:00 GMT

I expected NVLink to win this one.

In disaggregated serving on Dynamo, the decode engine pulls the KV cache out of the prefill engine through NIXL, and UCX picks the transport underneath. NVIDIA's own transport table puts NVLink at 450 to 900 GB/s and RoCE at 10 to 25 GB/s. So wherever NVLink can be used, it should be the faster road for the cache.

I tested it on a B300 node, same engines, same prompts, only the transport changed. RoCE moved the cache in 14 to 34 % less time. These are my notes on why.

Where NVLink is even an option

NVIDIA's guide draws the stack like this.

From NVIDIA's Disaggregated Inference Communication Guide (ai-dynamo/dynamo, Apache 2.0). Look at the transport layer: RDMA, host staging and TCP. NVLink is not on it.

NVLink is missing from that picture on purpose. The guide is about the operator, where prefill and decode are two pods, and it has a second diagram for why NVLink is not available between them.

From the same guide. NVLink works inside a pod, which is what tensor parallelism uses. Between two pods there is none: separate process namespaces, separate CUDA contexts.

So on the operator the question never comes up. Even with both pods on one node, the cache goes through the NIC.

From the same guide: two pods on the same node still use RDMA.

The question only comes up when Dynamo is started by hand, as plain processes with no pods. Then the prefill and the decode engine share one process namespace, UCX's cuda_ipc transport can open the other engine's GPU memory, and it rates itself far above the NIC. With UCX_TLS=rc_x,cuda_copy,cuda_ipc it wins the lane selection, and the KV cache moves GPU to GPU over NVLink with nothing on the rails.

That is the setup where I could put the two roads side by side.

The test


Hardware	One node, 8 B300 GPUs, NVLink between them, 400G-class RoCE NICs
Model	`google/gemma-4-31B-it`, BF16 weights and KV cache
Layout	1 prefill TP4 + 1 decode TP4 on the same node, started by hand
Software	Dynamo with vLLM 0.23 and NIXL 1.3.1
Load	4k, 8k and 16k-token prompts, 256 output tokens, 30 requests at a time
Without RoCE	`UCX_TLS=rc_x,cuda_copy,cuda_ipc`: UCX picks `cuda_ipc`, so NVLink
With RoCE	`UCX_TLS=rc_x,cuda_copy`: `cuda_ipc` removed, so `rc_mlx5` through the NIC

Same prompts in both runs. I checked the road each time with UCX_PROTO_INFO=y and with the RoCE port counters: without RoCE the NICs received 1.3 MB over the whole run, with RoCE they received 1,121 GB.

Time for one KV pull, per TP rank, 30 requests at a time

without RoCE (NVLink)with RoCE

4k-token prompts, 0.3 GB per pull

without RoCE23.9 ms

with RoCE15.7 ms

8k-token prompts, 0.4 GB per pull

without RoCE20.4 ms

with RoCE17.5 ms

16k-token prompts, 0.6 GB per pull

without RoCE29.2 ms

with RoCE21.4 ms

08162432 ms

Average over the pulls of each run. A request is four pulls, one per rank, at the same time. Linear scale from zero.

34 % less time at 4k, 14 % less at 8k, 27 % less at 16k. The road with more than ten times the bandwidth lost at every prompt length.

Why

NIXL reports two times per transfer: the whole transfer, and the post, which is how long it took just to issue the reads. Splitting the 16k pull that way shows where NVLink loses.

The same pull at 16k tokens, split in two

post time: issuing the readsafter the post, until the data has arrived

without RoCE11.7 + 17.5

with RoCE0.8 + 20.6

08162432 ms

Over NVLink, 11.7 ms goes to the post alone. Over RoCE the post is 0.8 ms.

The reason is the shape of the transfer. A KV pull is not one big copy. vLLM registers the cache per block and per layer, so one pull is a list of small descriptors: about 2,350 for a 4k prompt and 4,250 for a 16k prompt, around 125 KB each. NIXL's UCX backend issues one ucp_get_nbx per descriptor.

What one of those reads costs depends on the transport.

cuda_ipc. Each read becomes its own cuMemcpyDtoDAsync plus a cuEventRecord, issued one after another from a CPU thread. All the copies to one peer GPU go onto a single CUDA stream, so they run in order, and completions are collected 16 at a time. That is two driver calls and an event per 125 KB. Per descriptor, the post alone is 2.7 µs.

rc_mlx5. Each read becomes an RDMA READ work request on the NIC's send queue. The CPU queues all 4,250 in under a millisecond, 0.19 µs each, and from there the NIC does the work: zero-copy from GPU memory to GPU memory, striped over two paths.

So the NVLink run was never limited by NVLink. It moved about 19 GB/s per rank on a link that carries hundreds. It was limited by how fast one thread can push small copies through the CUDA driver.

Bandwidth is the wrong number to compare here. It is the same split as in storage, throughput against IOPS. This transfer is bound by operations, and RoCE has the cheaper operation because the NIC takes the whole batch off the CPU.

I think NVLink would win if the connector handed over a few large contiguous regions instead of thousands of blocks. I have not tested that.

Does it matter

	Without RoCE (NVLink)	With RoCE
KV pull time, 4k / 8k / 16k	1.0x	0.66x / 0.86x / 0.73x
Post time, 4k / 8k / 16k	1.0x	0.08x / 0.12x / 0.07x
TTFT p50, one request at a time	1.0x	0.99x / 1.00x / 1.00x
Requests per second, 30 at a time	1.0x	1.02x / 1.00x / 1.00x

Each figure is relative to the NVLink run at the same prompt length.

Not to the user. TTFT is the same to within 1 %, because a 20 ms pull is a small part of a wait that is mostly prefill. Throughput is the same too.

What it changes is how I think about the two links. NVLink is for the model: the TP ranks of one engine exchanging activations on every layer, with buffers set up once. RoCE is for moving the cache between engines. On the operator that is the only choice anyway, and after this test I would make the same choice by hand: drop cuda_ipc from UCX_TLS and let the NIC do it.

The case that does hurt is the one without RDMA at all. The same guide has the numbers for it.

Extra TTFT from the KV transfer, by transport (NVIDIA's numbers, log scale)

AWS EFA + libfabric, measured+37 ms

InfiniBand RDMA, expected+0.2 to 0.5 s

RoCE RDMA, expected+0.3 to 0.8 s

Staged through the host, expected+1 to 3 s

TCP fallback, measured+90 to 100 s

10 ms100 ms1 s10 s100 s

My chart of the performance table in NVIDIA's guide (Dynamo 1.4.2). Each bar ends at the top of its range, on a logarithmic scale. "Measured" rows are from an AWS p5.48xlarge with an 8B model and 8,000-token prompts; "expected" rows are the guide's estimate from hardware specs.

A first token after about 98 seconds on TCP, with nothing logged as an error. If a disaggregated deployment feels slow, check the transport before anything else.

NVIDIA Dynamo prom metrics patch

Fri, 02 Oct 2026 00:00:00 GMT

I serve an LLM with NVIDIA Dynamo. Its frontend exports latency metrics to Prometheus, and my dashboard said time to first token (TTFT) was about 2 seconds at p95.

What the dashboard could not tell me: is every request waiting 2 seconds, or only the ones with a long prompt? This post is about the small patch that answers that.

Dynamo can run prefill and decode on separate workers (disaggregated serving), which makes the question sharper: if TTFT is bad, I want to know whether to add prefill capacity, and for which kind of request.

The problem: one number for every prompt size

Dynamo exports a good set of metrics out of the box. NVIDIA lists them all in the Metrics Catalog. They have one gap.

Look at the labels on the latency histograms (TTFT, ITL, request duration). They carry model, and nothing about the request. Input length is exported too, but as its own separate histogram.

You cannot join two histograms. A Prometheus histogram is a set of counters ("how many observations were at most 0.5 s"), with no row per request. The TTFT histogram knows 900 requests came back in under half a second. The ISL histogram knows 4,000 prompts were under 1k tokens. Nothing records which were which.

So a 300-token chat turn and a 12,000-token RAG prompt go into the same TTFT histogram, even though prefill for the second one is far more work. When p95 moves, I cannot tell whether the system got slower or the traffic just shifted toward longer prompts. And I cannot tell which context length is the bottleneck.

Here is the same data both ways. Start on Stock, then switch to Patched.

One latency number, or one per input-length bucket

About 18,000 requests from load tests against a 31B model on a test cluster, one prefill and one decode worker. Faded bars are buckets with only 16 requests each, so treat those as noise. Hover or tab to a bar for the request count.

Table view

Input length (tokens)	Requests	TTFT p50	TTFT p95	TTFT p99	ITL p50	ITL p95	Avg cached tokens
all requests	17,944	0.17 s	2.00 s	5.29 s	7.3 ms	12.5 ms
0-512	4,700	0.07 s	0.19 s	0.22 s	6.4 ms	12.3 ms	1
513-1024	16	0.08 s	0.12 s	0.20 s	6.3 ms	12.3 ms	0
1025-2048	16	0.08 s	0.20 s	0.22 s	6.7 ms	12.4 ms	0
2049-4096	118	0.23 s	1.32 s	2.02 s	9.3 ms	12.7 ms	0
4097-8192	247	0.40 s	3.98 s	4.56 s	9.3 ms	12.6 ms	30
8193-16384	1,067	0.58 s	5.90 s	9.18 s	8.1 ms	12.5 ms	8,745
16385+	11,780	0.31 s	2.06 s	5.30 s	7.4 ms	12.5 ms	21,074

The patch: an `isl_bucket` label

The fix is to label the latency histograms with the bucket the prompt falls in.

It is a small patch to the Dynamo frontend, on top of Dynamo 1.5.0. In general terms it does three things:

Adds one opt-in setting on the frontend: a list of bucket boundaries, as inclusive upper bounds in tokens. Six boundaries make seven buckets, and each bucket becomes a label value: 0-512, 513-1024, and so on up to the open-ended 16385+.
Exports five more histograms next to the originals: TTFT, ITL, request duration, output length and cached tokens, each with an isl_bucket label beside model. They use the same le buckets as the originals, so a quantile from one is comparable to a quantile from the other.
Creates every bucket's series at zero when the model registers, so rate() and increase() do not miss a bucket's first burst of requests.

When does the frontend know the bucket?

At the first token. That is the point where the frontend has the prompt's token count and records TTFT, so the request's bucket is latched there and reused for everything after:

TTFT is observed straight away, into the bucket's series.
ITL goes through a request-local histogram, the same way the stock ITL metric does, so the per-token path stays cheap.
Request duration, OSL and cached tokens are recorded when the request ends.

A request that fails before its first token never gets a bucket. It still shows up in dynamo_frontend_requests_total.

What it costs

Every bucket multiplies the series of five histograms. For one model and seven buckets that came to about 270 extra series per frontend, which is nothing for Prometheus. The patch refuses more than 32 boundaries so a typo cannot blow up cardinality.

When the setting is off, nothing extra is registered and requests pay nothing for it.

Querying it

p95 TTFT per bucket:

histogram_quantile(0.95,
  sum by (le, isl_bucket) (
    rate(dynamo_frontend_time_to_first_token_by_isl_seconds_bucket[5m])
  )
)

Mean TTFT per bucket:

  sum by (isl_bucket) (rate(dynamo_frontend_time_to_first_token_by_isl_seconds_sum[5m]))
/
  sum by (isl_bucket) (rate(dynamo_frontend_time_to_first_token_by_isl_seconds_count[5m]))

Without and with, on a real dashboard

The same Grafana panel, TTFT p95, over the same ten minutes of a load test.

Without the label, which is all stock Dynamo can give you:

One line. It climbs from about 2 s to almost 10 s, and all I can say is that it got slow.

With the isl_bucket label:

Five lines, and now there is something to act on. Prompts of 2k to 4k tokens stayed around 1.3 s the whole time. The 4k to 8k bucket settled near 4.4 s. Everything above 8k tokens went to almost 10 s, and that is what dragged the single line up.

Two things I learned building the dashboard:

Bucket labels sort as strings, so 1025-2048 comes before 513-1024. On VictoriaMetrics, wrapping the query in sort_by_label_numeric(..., "isl_bucket") puts them in order.
Give avg, p50, p95 and p99 their own panels. On one shared panel the p99 spikes set the y-axis and flatten everything else into a line along the bottom.

Conclusion

These metrics are worth having. With one latency number, a slow dashboard only tells me that something is slow. With one number per prompt-length bucket, when a bottleneck shows up, at least I know which bottleneck it is: which prompt sizes are waiting, and so where the capacity is short.

NVFP4 eval bench on B300

Fri, 02 Oct 2026 00:00:00 GMT

I had B300 GPUs to work with (NVIDIA's Blackwell generation), and NVIDIA publishes a 4-bit version of a model I serve: nvidia/Gemma-4-31B-IT-NVFP4, next to the normal google/gemma-4-31B-it.

On paper it is an easy choice if you have Blackwell: the NVFP4 checkpoint is about half the size (33 GB against 62 GB) and it serves faster. The open question is quality: does the model get dumber? So before using it, I ran an eval benchmark on both models and compared them.

The experiment

Both models ran on the same machine, the same serving software and the same flags. Only the weights changed.

One node with 8 NVIDIA B300 GPUs.
vLLM 0.28 behind NVIDIA Dynamo: one prefill worker and one decode worker, 4 GPUs each.
65k context window.

Only one model fits at a time, so I ran every test on BF16, swapped the weights, and ran every test again. The comparison is BF16 against the checkpoint as NVIDIA ships it.

Three kinds of eval

Eval	What it asks	How it is scored	Size
Multi-turn function calling	Can the model pick and call the right tool across a long conversation?	Tool-call F1	100 conversations, about 20 turns each
GSM8K	Grade-school maths word problems	The last number in the reply must match	1319 questions
Multilingual MMLU (`openai/MMMLU`)	General knowledge, multiple choice, in French, German, Spanish, Chinese and Japanese	The reply must be the right letter	200 per language, 1000 in total

Function calling is the one closest to real work. Each conversation has more than 40 tools and is written in Manglish. The eval replays it turn by turn:

The user message goes in.
The model replies with a tool call or with text.
The reply is compared with the reference.
The reference tool result is injected, because there is no real backend.
The next user message goes in.

The headline number is tool-call F1: did the model call a tool when it should, and stay quiet when it should not. The eval also reports whether it picked the right function, whether the arguments are valid JSON, whether it invented a function, and whether the required arguments are there.

GSM8K asks the model to end its reply with #### <number>. MMMLU asks for a single letter. Both are easy to score and hard to argue with.

The function-calling set is synthetic: the conversations were written by LLMs for our own use case. So read its score as a comparison between the two models on the same test, not as an absolute score.

Telling a real drop from noise

There are two situations, and they need different handling.

When the model samples, repeat. The function-calling eval sends no temperature, so the model uses its own default (temperature 1.0). The same model on the same test gave three different scores: 0.950, 0.946, 0.944. So each model ran 3 times and I report the mean and the spread. A gap only counts if it is clearly bigger than that spread.

Tool-call F1 of each run, 3 runs per model

BF16

mean 0.947

NVFP4

mean 0.945

0.9400.9440.9480.952

Each dot is one full pass over the 100 conversations. The grey bar is the mean. The two models overlap, so the 0.002 gap between their means is noise. The axis is zoomed in: it spans only 0.012.

When the model is deterministic, pair the questions. GSM8K and MMMLU run at temperature 0, so one run is enough. Here I compare question by question, and count two things: the questions only BF16 got right, and the questions only NVFP4 got right. If the 4-bit model is no worse, the two counts should be about equal. McNemar's exact test puts a p-value on that.

Results

Eval	BF16	NVFP4	Real difference?
Function calling, tool-call F1 (3 runs each)	0.947 ± 0.003	0.945 ± 0.002	No. The gap is smaller than the run-to-run spread.
GSM8K (1319)	0.970	0.971	No. Only 2 questions flipped one way and 3 the other.
Multilingual MMLU (1000)	0.801	0.779	Yes. 46 questions only BF16 got right, 24 only NVFP4 got right, p = 0.012.

Score on each eval, BF16 against NVFP4

BF16NVFP4

Function callingtool-call F1

0.947

0.945

GSM8K1319 questions

0.970

0.971

Multilingual MMLU1000 questions

0.801

0.779

Bars start at 0 and a full-width bar would be 1.0. Only the multilingual MMLU pair differs by more than noise.

Multilingual MMLU by language, 200 questions each

BF16NVFP4

French

0.815 → 0.820

German

0.820 → 0.800

Spanish

0.795 → 0.765

Chinese

0.785 → 0.760

Japanese

0.790 → 0.750

0.700.750.800.85

The line between the dots is what the 4-bit model lost or gained. The axis is zoomed in, from 0.70 to 0.85.

The 4-bit model holds on tool calling and on maths. It loses about 2 points of general knowledge across the five languages. The loss sits mostly in Spanish, Chinese and Japanese. French did not drop at all.

As a side effect, NVFP4 was faster on the same work: 29% more output tokens per second on the GSM8K run, at 32 requests at a time. That number comes from an accuracy run, not from a speed benchmark.

Conclusion

NVFP4 keeps the scores that matter for my use: tool calling and maths are the same as BF16, within noise. The cost is about 2 points of multilingual general knowledge. For that, the model is half the size and faster, so on Blackwell it is worth using. If your work leans on general knowledge in other languages, run the eval on your own data first.

Ariff Nazhan

KAI Scheduler vs HAMi

What each one is

What each one is good at

Things to know before choosing

Why we use HAMi

When KAI is the better choice

Proxmox with GPU passthrough

The PC

Installing Proxmox: keep host drivers off the GPU

The plan: three VMs

Step 1: the Ubuntu ISO

Step 2: two plain VMs

Step 3: the GPU VM

What passthrough needs

The VM settings

Attaching the card

The GPU VM takes the monitor

Step 4: install Ubuntu

Step 5: fixed addresses on the router

Check: the card inside the VM

What it looks like at the end

RoCE vs NVLink for KV transfer

Where NVLink is even an option

The test

Why

Does it matter

NVIDIA Dynamo prom metrics patch

The problem: one number for every prompt size

The patch: an isl_bucket label

When does the frontend know the bucket?

What it costs

Querying it

Without and with, on a real dashboard

Conclusion

NVFP4 eval bench on B300

The experiment

Three kinds of eval

Telling a real drop from noise

Results

Conclusion

The patch: an `isl_bucket` label