RoCE vs NVLink for KV transfer
Credits to Scicom (MSC) Berhad and Scicom AI Enterprise for the resources.
I expected NVLink to win this one.
In disaggregated serving on Dynamo, the decode engine pulls the KV cache out of the prefill engine through NIXL, and UCX picks the transport underneath. NVIDIA's own transport table puts NVLink at 450 to 900 GB/s and RoCE at 10 to 25 GB/s. So wherever NVLink can be used, it should be the faster road for the cache.
I tested it on a B300 node, same engines, same prompts, only the transport changed. RoCE moved the cache in 14 to 34 % less time. These are my notes on why.
Where NVLink is even an option
NVIDIA's guide draws the stack like this.
NVLink is missing from that picture on purpose. The guide is about the operator, where prefill and decode are two pods, and it has a second diagram for why NVLink is not available between them.
So on the operator the question never comes up. Even with both pods on one node, the cache goes through the NIC.
The question only comes up when Dynamo is started by hand, as plain processes with no pods. Then the prefill and the decode engine share one process namespace, UCX's cuda_ipc transport can open the other engine's GPU memory, and it rates itself far above the NIC. With UCX_TLS=rc_x,cuda_copy,cuda_ipc it wins the lane selection, and the KV cache moves GPU to GPU over NVLink with nothing on the rails.
That is the setup where I could put the two roads side by side.
The test
| Hardware | One node, 8 B300 GPUs, NVLink between them, 400G-class RoCE NICs |
| Model | google/gemma-4-31B-it, BF16 weights and KV cache |
| Layout | 1 prefill TP4 + 1 decode TP4 on the same node, started by hand |
| Software | Dynamo with vLLM 0.23 and NIXL 1.3.1 |
| Load | 4k, 8k and 16k-token prompts, 256 output tokens, 30 requests at a time |
| Without RoCE | UCX_TLS=rc_x,cuda_copy,cuda_ipc: UCX picks cuda_ipc, so NVLink |
| With RoCE | UCX_TLS=rc_x,cuda_copy: cuda_ipc removed, so rc_mlx5 through the NIC |
Same prompts in both runs. I checked the road each time with UCX_PROTO_INFO=y and with the RoCE port counters: without RoCE the NICs received 1.3 MB over the whole run, with RoCE they received 1,121 GB.
Time for one KV pull, per TP rank, 30 requests at a time
without RoCE (NVLink)with RoCE
4k-token prompts, 0.3 GB per pull
8k-token prompts, 0.4 GB per pull
16k-token prompts, 0.6 GB per pull
34 % less time at 4k, 14 % less at 8k, 27 % less at 16k. The road with more than ten times the bandwidth lost at every prompt length.
Why
NIXL reports two times per transfer: the whole transfer, and the post, which is how long it took just to issue the reads. Splitting the 16k pull that way shows where NVLink loses.
The same pull at 16k tokens, split in two
post time: issuing the readsafter the post, until the data has arrived
The reason is the shape of the transfer. A KV pull is not one big copy. vLLM registers the cache per block and per layer, so one pull is a list of small descriptors: about 2,350 for a 4k prompt and 4,250 for a 16k prompt, around 125 KB each. NIXL's UCX backend issues one ucp_get_nbx per descriptor.
What one of those reads costs depends on the transport.
cuda_ipc. Each read becomes its own cuMemcpyDtoDAsync plus a cuEventRecord, issued one after another from a CPU thread. All the copies to one peer GPU go onto a single CUDA stream, so they run in order, and completions are collected 16 at a time. That is two driver calls and an event per 125 KB. Per descriptor, the post alone is 2.7 µs.
rc_mlx5. Each read becomes an RDMA READ work request on the NIC's send queue. The CPU queues all 4,250 in under a millisecond, 0.19 µs each, and from there the NIC does the work: zero-copy from GPU memory to GPU memory, striped over two paths.
So the NVLink run was never limited by NVLink. It moved about 19 GB/s per rank on a link that carries hundreds. It was limited by how fast one thread can push small copies through the CUDA driver.
Bandwidth is the wrong number to compare here. It is the same split as in storage, throughput against IOPS. This transfer is bound by operations, and RoCE has the cheaper operation because the NIC takes the whole batch off the CPU.
I think NVLink would win if the connector handed over a few large contiguous regions instead of thousands of blocks. I have not tested that.
Does it matter
| Without RoCE (NVLink) | With RoCE | |
|---|---|---|
| KV pull time, 4k / 8k / 16k | 1.0x | 0.66x / 0.86x / 0.73x |
| Post time, 4k / 8k / 16k | 1.0x | 0.08x / 0.12x / 0.07x |
| TTFT p50, one request at a time | 1.0x | 0.99x / 1.00x / 1.00x |
| Requests per second, 30 at a time | 1.0x | 1.02x / 1.00x / 1.00x |
Each figure is relative to the NVLink run at the same prompt length.
Not to the user. TTFT is the same to within 1 %, because a 20 ms pull is a small part of a wait that is mostly prefill. Throughput is the same too.
What it changes is how I think about the two links. NVLink is for the model: the TP ranks of one engine exchanging activations on every layer, with buffers set up once. RoCE is for moving the cache between engines. On the operator that is the only choice anyway, and after this test I would make the same choice by hand: drop cuda_ipc from UCX_TLS and let the NIC do it.
The case that does hurt is the one without RDMA at all. The same guide has the numbers for it.
Extra TTFT from the KV transfer, by transport (NVIDIA's numbers, log scale)
A first token after about 98 seconds on TCP, with nothing logged as an error. If a disaggregated deployment feels slow, check the transport before anything else.