⚡ d3-lite on the NPU

Accuracy and latency of d3:lite (vllm-sr/d3-lite v3.1.3, package D3-Lite-NPU2) served by FastFlowLM 1.0.11.

Note:

• AMD Ryzen AI MAX+ PRO 395 (Strix Halo, XDNA 2 NPU), 128 GB, Ubuntu 24.04, XRT, FLM’s default NPU power mode (Performance).
• Latency is per request at the engine (no HTTP), median of 20 runs after warm-up; HTTP adds about a millisecond.

Accuracy against the upstream BF16 model

Paired gold accuracy, upstream BF16 minus NPU: +0.69 pt (95% CI [+0.10, +1.28] pt). Prompts, token ids and answer objects are byte-identical to the upstream runtime.

Latency

First request after flm serve is ready: 191 ms (steady state 168 ms over HTTP).

Rows of 2,048 tokens and more: median of 15 runs (32,768: 3). From 1,536 tokens attention runs on the NPU (below that the CPU is faster); the first such request also loads the attention kernel (about 0.1 s) and from then on the decision model holds two NPU hardware contexts instead of one.

Images

Accuracy on the Decision Index public vision suite (the 10 benchmarks that can be rebuilt without gated data; 60 questions each, images capped at 1.6 MP as the suite’s decide engine sends them):

Paired gold accuracy, NPU minus upstream BF16: +1.50 pt (95% CI [−0.92, +3.92] pt). Image grids and input token counts equal upstream’s on every question; the processor’s pixels are within 2 grey levels of upstream’s.

Latency with and without an image, at the same total input length (one question, over HTTP, median of 5; 8,192 tokens: 3 runs; 16,384 and 32,000: 1):

An image costs a fixed amount on top of its tokens (decoding, resizing and the vision tower): about 0.2 s at 640 × 480 and 0.8 s at 1.6 MP. All questions of a request share one image encoding: a 1.6 MP image with 8 questions takes 6.4 s, not 8 × 1.5 s. The first request with images loads the vision tower (about 40 ms) and two more NPU hardware contexts.

⚡ d3-nano on the NPU

Accuracy and latency of d3:nano (vllm-sr/d3-nano v3.1.3, package D3-Nano-NPU2), same machine and settings as above.

Note:

• Latency is per request at the engine (no HTTP), median of 3 runs in each of 2 processes; the NPU is bimodal per process (about ±15%).
• Prompts are token ids of the given length. Rows of 7,869 tokens and more are real documentation text, median of 3 requests (32,768: 1 request).

Accuracy against the upstream BF16 model

Prompts, token ids and answer objects follow the upstream runtime. On the Decision Index sample the mean benchmark score is 1.25 points below the upstream BF16 model (standard error 0.58; about 65 to 80 requests per benchmark, so a single benchmark is noisy). This is a sample, not the full suite, and a different one from the d3-lite subset. The vision subset leaves out Winoground (gated), as for d3-lite; paired accuracy, BF16 minus d3-nano, is +0.3 points (95% CI -2.3 to +2.8). The GPTQ calibration prompts (300 synthetic questions) contain no Decision Index data.

Images (d3-nano)

Image requests agree with the upstream BF16 model on 56 of 60 answers (93.3%; 31 hand-built requests with 64 × 64 to 2000 × 1500 images, one to several images each; median probability difference 0.026). Latency over HTTP, one question, median of 3: 1.3 s with a 1,024 × 768 image (876 input tokens), 2.4 s with two images, 3.0 s with a 2000 × 1500 image (downscaled to the 1.6 MP budget). A request with 8 questions about one image takes 5.0 s (one image encoding). PNG, JPEG and WebP data URLs are accepted; web addresses, file paths, corrupt images and videos are refused with HTTP 422.

Latency

Attention runs on the NPU from 1,536 tokens, as for d3-lite (measured break-even for d3-nano: about 1,550 tokens). At 31,491 tokens that is 3.3× faster than attention on the CPU (27 s against 91 s) with no measurable change in accuracy. A prompt of 32,769 tokens is refused.

⚡ d3-mini on the NPU

Accuracy and latency of d3:mini (vllm-sr/d3-mini v3.1.3, package D3-Mini-NPU2), same machine and settings as above.

Note:

• Latency is per request at the engine (no HTTP), median of 3 runs in each of 2 processes; the NPU is bimodal per process (about ±15%). Rows of 16,116 tokens and more are real documentation text, one request each.

Accuracy against the upstream BF16 model

Prompts, token ids and answer objects follow the upstream runtime. On the Decision Index sample the mean benchmark score is 0.99 points below the upstream BF16 model (standard error 0.57; about 37 requests per benchmark, so a single benchmark is noisy). This is a sample, not the full suite, and the same sampling as for d3-nano, not the d3-lite subset. The vision subset leaves out Winoground (gated), as for d3-lite and d3-nano; paired accuracy, BF16 minus d3-mini, is +2.5 points (95% CI +0.5 to +4.5): images cost a little accuracy (largest gaps on Mind2Web, MMMU-Pro, CV-Bench and CharXiv, about 60 questions each), while the answers agree with BF16 on 89.8% of the questions. The GPTQ calibration prompts (300 synthetic questions) contain no Decision Index data.

Images (d3-mini)

Image requests agree with the upstream BF16 model on 58 of 60 answers (96.7%; 31 hand-built requests with 64 × 64 to 2000 × 1500 images, one to several images each; median probability difference 0.024). Latency over HTTP, one question, median of 3: 2.1 s with a 1,024 × 768 image (876 input tokens), 3.8 s with two images, 4.4 s with a 2000 × 1500 image (downscaled to the 1.6 MP budget). A request with 8 questions about one image takes 10.8 s (one image encoding). PNG, JPEG and WebP data URLs are accepted; web addresses, file paths, corrupt images and videos are refused with HTTP 422.

Latency

Attention runs on the NPU from 1,536 tokens, as for d3-lite (measured break-even for d3-mini: about 1,536 tokens; at 4,096 tokens the NPU attention makes a request 17% faster, 5.3 s against 6.3 s). A prompt of 32,769 tokens is refused.