⚡ Performance and Efficiency Benchmarks

This section reports the performance on NPU with FastFlowLM (FLM).

Note:

  • The 1k, 2k and 4k columns are based on FastFlowLM v1.0.13 (Q4_K weights); the 8k and 16k columns are still from v1.0.5 (Q4_0 weights) and are not directly comparable.
  • Under FLM’s default NPU power mode (Performance)
  • Newer versions may deliver improved performance.
  • Hy-MT2 is a dedicated translation model tuned for short, single-turn requests, so it is benchmarked up to 16k max context length rather than the longer sweeps used for general chat models.

Test System:

AMD Ryzen™ AI 9 370 (Strix Point) with 32 GB DRAM; performance is comparable to other Strix Point and Strix Halo systems.


🚀 Decoding Speed (TPS, or Tokens per Second, starting @ different context lengths)

Model HW 1k 2k 4k 8k 16k
Hy-MT2-1.8B XDNA2 43.2 41.5 38.9 28.4 19

🚀 Prefill Speed (TPS, or Tokens per Second, with different prompt lengths)

Model HW 1k 2k 4k 8k 16k
Hy-MT2-1.8B XDNA2 1,701 1,574 1,388 912 723