Ariff Nazhan

Article

NVFP4 eval bench on B300

Credits to Scicom (MSC) Berhad and Scicom AI Enterprise for the resources.

I had B300 GPUs to work with (NVIDIA's Blackwell generation), and NVIDIA publishes a 4-bit version of a model I serve: nvidia/Gemma-4-31B-IT-NVFP4, next to the normal google/gemma-4-31B-it.

On paper it is an easy choice if you have Blackwell: the NVFP4 checkpoint is about half the size (33 GB against 62 GB) and it serves faster. The open question is quality: does the model get dumber? So before using it, I ran an eval benchmark on both models and compared them.

The experiment

Both models ran on the same machine, the same serving software and the same flags. Only the weights changed.

  • One node with 8 NVIDIA B300 GPUs.
  • vLLM 0.28 behind NVIDIA Dynamo: one prefill worker and one decode worker, 4 GPUs each.
  • 65k context window.

Only one model fits at a time, so I ran every test on BF16, swapped the weights, and ran every test again. The comparison is BF16 against the checkpoint as NVIDIA ships it.

Three kinds of eval

Eval What it asks How it is scored Size
Multi-turn function calling Can the model pick and call the right tool across a long conversation? Tool-call F1 100 conversations, about 20 turns each
GSM8K Grade-school maths word problems The last number in the reply must match 1319 questions
Multilingual MMLU (openai/MMMLU) General knowledge, multiple choice, in French, German, Spanish, Chinese and Japanese The reply must be the right letter 200 per language, 1000 in total

Function calling is the one closest to real work. Each conversation has more than 40 tools and is written in Manglish. The eval replays it turn by turn:

  1. The user message goes in.
  2. The model replies with a tool call or with text.
  3. The reply is compared with the reference.
  4. The reference tool result is injected, because there is no real backend.
  5. The next user message goes in.

The headline number is tool-call F1: did the model call a tool when it should, and stay quiet when it should not. The eval also reports whether it picked the right function, whether the arguments are valid JSON, whether it invented a function, and whether the required arguments are there.

GSM8K asks the model to end its reply with #### <number>. MMMLU asks for a single letter. Both are easy to score and hard to argue with.

The function-calling set is synthetic: the conversations were written by LLMs for our own use case. So read its score as a comparison between the two models on the same test, not as an absolute score.

Telling a real drop from noise

There are two situations, and they need different handling.

When the model samples, repeat. The function-calling eval sends no temperature, so the model uses its own default (temperature 1.0). The same model on the same test gave three different scores: 0.950, 0.946, 0.944. So each model ran 3 times and I report the mean and the spread. A gap only counts if it is clearly bigger than that spread.

Tool-call F1 of each run, 3 runs per model

BF16
mean 0.947
NVFP4
mean 0.945
0.9400.9440.9480.952
Each dot is one full pass over the 100 conversations. The grey bar is the mean. The two models overlap, so the 0.002 gap between their means is noise. The axis is zoomed in: it spans only 0.012.

When the model is deterministic, pair the questions. GSM8K and MMMLU run at temperature 0, so one run is enough. Here I compare question by question, and count two things: the questions only BF16 got right, and the questions only NVFP4 got right. If the 4-bit model is no worse, the two counts should be about equal. McNemar's exact test puts a p-value on that.

Results

Eval BF16 NVFP4 Real difference?
Function calling, tool-call F1 (3 runs each) 0.947 ± 0.003 0.945 ± 0.002 No. The gap is smaller than the run-to-run spread.
GSM8K (1319) 0.970 0.971 No. Only 2 questions flipped one way and 3 the other.
Multilingual MMLU (1000) 0.801 0.779 Yes. 46 questions only BF16 got right, 24 only NVFP4 got right, p = 0.012.

Score on each eval, BF16 against NVFP4

BF16NVFP4

Function callingtool-call F1
0.947
0.945
GSM8K1319 questions
0.970
0.971
Multilingual MMLU1000 questions
0.801
0.779
Bars start at 0 and a full-width bar would be 1.0. Only the multilingual MMLU pair differs by more than noise.

Multilingual MMLU by language, 200 questions each

BF16NVFP4

French
0.815 → 0.820
German
0.820 → 0.800
Spanish
0.795 → 0.765
Chinese
0.785 → 0.760
Japanese
0.790 → 0.750
0.700.750.800.85
The line between the dots is what the 4-bit model lost or gained. The axis is zoomed in, from 0.70 to 0.85.

The 4-bit model holds on tool calling and on maths. It loses about 2 points of general knowledge across the five languages. The loss sits mostly in Spanish, Chinese and Japanese. French did not drop at all.

As a side effect, NVFP4 was faster on the same work: 29% more output tokens per second on the GSM8K run, at 32 requests at a time. That number comes from an accuracy run, not from a speed benchmark.

Conclusion

NVFP4 keeps the scores that matter for my use: tool calling and maths are the same as BF16, within noise. The cost is about 2 points of multilingual general knowledge. For that, the model is half the size and faster, so on Blackwell it is worth using. If your work leans on general knowledge in other languages, run the eval on your own data first.