⚡ CLI Mode

FLM CLI mode offers a familiar terminal-based interactive experience, fully offline and accelerated exclusively on AMD NPUs. Here are detailed descriptions of commands and setup for CLI mode usage. It includes:

• 🔧 Pre-Run PowerShell Commands (System)
• 💻 Commands Inside CLI Mode
• 🗂️ Others

🔧 Pre-Run PowerShell Commands (System)

🖥️ System Compatibility Check

Verify that your hardware meets the necessary requirements to run FastFlowLM:

flm validate

Output the validation results as a JSON object:

flm validate --json

🆘 Show Help

flm help

🚀 Run a Model

Run a model interactively from the terminal:

flm run llama3.2:1b

flm is short for FastFlowLM. If the model isn’t available locally, it will be downloaded automatically. This launches FastFlowLM in CLI mode.

Linux note: flm validate checks the kernel DRM device, while flm run opens the NPU through XRT. If validation succeeds but flm run fails with No such device with index '0', confirm XRT can see the NPU:

xrt-smi examine

On Arch Linux, install xrt-plugin-amdxdna in addition to xrt and the amdxdna driver. See the Linux install guide for the full driver and firmware checklist.

⬇️ Pull a Model (Download Only)

Download a model from HuggingFace without launching it:

flm pull llama3.2:3b

This code forces a re-download of the model, overwriting the current version.

flm pull llama3.2:3b --force

⚠️ Use --force only if the model file is corrupted (e.g., incomplete download). Proceed with caution.

📦 Some tags are built on another one: nemotron3.5-bolt-dflash:e4b adds a small draft package to the nemotron3.5-bolt:e4b weights. flm pull downloads the base package first (from the same source, --modelscope included), flm list marks such a tag “(requires …)”, and it is ready only when both packages are.

📁 Default Model Storage Location

🔧 Changing the Model Storage Path

You can override the default location by setting the FLM_MODEL_PATH environment variable.

Windows — Update the existing system environment variable:

1. Open Start and search for “Edit the system environment variables”.
2. Click Environment Variables….
3. Under System variables, find FLM_MODEL_PATH, select it, and click Edit….
4. Update the value to your desired path (e.g., D:\models\flm).
5. Click OK and restart any open terminals for the change to take effect.

Linux — Set temporarily for the current shell session:

export FLM_MODEL_PATH="/your/custom/path"

To make the change permanent, add the line above to your ~/.bashrc, then reload it:

echo 'export FLM_MODEL_PATH="/your/custom/path"' >> ~/.bashrc
source ~/.bashrc

📦 List Supported and Downloaded Models

Display all available models and locally downloaded models:

flm list

Output as JSON:

flm list --json

Filters flag:

# Show everything
flm list --filter all

# Only models already installed
flm list --filter installed

# Only models not yet installed
flm list --filter not-installed

Quiet mode:

# Default view (pretty, with icons)
flm list

# Quiet view (no emoji / minimal)
flm list --quiet

# Show everything
flm list --filter all --quiet

# Only models already installed
flm list --filter installed --quiet

# Only models not yet installed
flm list --filter not-installed --quiet

❌ Remove a Downloaded Model

Delete a model from local storage:

flm remove llama3.2:3b

For a tag built on another one (for example nemotron3.5-bolt-dflash:e4b), flm remove deletes only its own package; the base model stays. Removing a base model that other installed tags need prints a warning naming them.

✅ Check a Downloaded Model

Verify the file hashes for a downloaded model:

flm check llama3.2:3b

🚀 Start Server Mode (Local)

Launch FastFlowLM as a local REST API server (also supports the OpenAI API):

flm serve llama3.2:1b

🔌 Show Server Port

Show current FLM port (default) in PowerShell:

flm port

⚡ NPU Power Mode

By default, FLM runs in performance NPU power mode. You can switch to other NPU power modes (powersaver, balanced, or turbo) using the --pmode flag:

For CLI mode:

flm run gemma3:4b --pmode balanced

For Server mode:

flm serve gemma3:4b --pmode balanced

📏 Set Context Length at Launch

The default context length for each model can be found here.

Set the context length with --ctx-len (or -c).

In PowerShell, run:

For CLI mode:

flm run llama3.2:1b --ctx-len 8192

For Server mode:

flm serve llama3.2:1b --ctx-len 8192

• Internally, FLM enforces a minimum context length of 512. If you specify a smaller value, it will automatically be adjusted up to 512.
• If you enter a context length that is not a power of 2, FLM automatically rounds it up to the nearest power of 2. For example: input 8000 → adjusted to 8192.

🔀 Choose an Execution Backend

A backend names where the kernels come from. There are two:

You normally never set this. The model tag already names the flow — a family ending in -core wants corelib’s kernels, everything else FastFlowLM’s — and flm list only ever offers you tags this build and this machine can actually run. flm run separately prints the silicon as NPU platform: dev04. A model family has at most one engine per backend, so there is nothing to choose between.

The flag exists for overriding what the tag says, and for the targets that will join this list later:

flm run   phi4-mini-it:4b     --backend flm
flm serve phi4-mini-it-core:4b --backend core

Precedence, highest first:

A per-request "backend" overrides --backend for that request, and reloads the model if it differs from the one already loaded — exactly as asking for a different model does.

Asking for a backend this build does not ship fails at load with a message listing what is actually available. There is no silent fallback to another engine or to the CPU. Note that a release is built for one NPU generation: on hardware it was not built for, run, serve and bench refuse up front, and flm validate still reports what it found.

A build carries the engines for one generation only, so there is no flag that moves it to the other catalog entry — that is a different build of FLM.

🖧 Set Server Port at Launch

Set a custom port at launch:

  flm serve llama3.2:1b --port 8000
  flm serve llama3.2:1b -p 8000

⚠️ --port (-p) only affects the current run; it won’t change the default port.

🛠️ Set Host at Launch

Specify a custom host address when starting the server:

flm serve llama3.2:1b --host 127.0.0.1

⚠️ Note: –host applies only to the current session. It does not modify the default host configuration (default: 127.0.0.1).

🌐 Cross-Origin Resource Sharing (CORS)

CORS lets browser apps hosted on a different origin call your FLM server safely.

• Enable CORS

flm serve --cors 1

• Disable CORS

flm serve --cors 0

⚠️ Default: CORS is enabled.
🔒 Security tip: Disable CORS (or restrict at your proxy) if your server is exposed beyond localhost (127.0.0.1).

⏸️ Preemption

Preemption allows high-priority tasks to interrupt ongoing NPU jobs, improving responsiveness for critical workloads. To enable preemption:

For CLI mode:

flm run llama3.2:1b --preemption 1

For Server mode:

flm serve llama3.2:1b --preemption 1

⚠️ Note: Preemption is for engineering testing/optimization only. It requires a special driver + toolkit and is not for public use.

🧩 Change Prefill Chunk Size at Launch

The --prefill-chunk-len flag controls how many tokens are processed per chunk during the prefill phase of inference. The default is the model’s own max_prefill_len: 4096 for most models, 1024 for the flash models.

For CLI mode:

flm run llama3.2:1b --prefill-chunk-len 8192

For Server mode:

flm serve llama3.2:1b --prefill-chunk-len 8192

• Values below 512 are raised to 512, and the value is rounded up to a power of 2 (5000 → 8192).
• The chunk length is not a limit on the prompt: a longer prompt is read in more chunks. The prompt has to fit in the context length (--ctx-len). See Long Prompts: Prefill Chunking.

🗃️ Prefix Cache (Server Mode)

In Server mode, Qwen3.5 (0.8b, 2b, 4b, 9b) and qwen3.6-moe:35b-a3b remember the system prompt, tools and earlier turns they have already read, so an agent that sends the same long prompt again only waits for what is new. It is on by default.

flm serve qwen3.5:4b --prefix-cache-mb 8192 --prefix-slots 16

flm run (CLI mode) does not use it. See Prefix Cache (Prompt Reuse) for memory per model, how to check that it works, and its limits.

🎙️ ASR (Automatic Speech Recognition)

Requirement: The ASR model (e.g., whisper-large-v3-turbo) must run with an LLM loaded concurrently. Enabling --asr 1 starts Whisper in the background while your chosen LLM loads.

CLI mode

flm run gemma3:4b --asr 1  # Load Whisper (whisper-large-v3-turbo) in the background and load the LLM (gemma3:4b) concurrently.

Server mode

flm serve gemma3:4b --asr 1  # Background-load Whisper and initialize the LLM (gemma3:4b) concurrently.

Note: ASR alone isn’t supported—an LLM must be present for end-to-end voice→text→LLM workflows.

See the ASR guide here

🧭 Decision Model (System One)

--decision <tag> loads a decision model (d3:lite) next to the chat model, or alone, and serves POST /v1/systemone (text and images, up to 32,768 input tokens per question; FastFlowLM 1.0.11 or later). Decision models run in Server Mode only (flm run d3:lite prints how to serve it).

flm serve --decision d3:lite                 # decision model only
flm serve qwen3.5:4b --decision d3:lite      # next to a chat model

See the d3 guide here

💻 Commands Inside CLI Mode

Once inside the CLI, use the following commands. System commands always start with / (e.g., /help).

🆘 Help

/?

Displays all available interactive system commands. Highly recommended for first-time users.

🪪 Model Info

/show

View model architecture, size, max context size (Adjustable – see bottom) and more.

🔄 Change Model

/load [model_name]

Unload the current model and load a new one. KV cache will be cleared.

💾 Save Conversation

/save

Save the current conversation history to disk.

🧹 Clear Memory

/clear

Clear the KV cache (model memory) for a fresh start.

📊 Show Runtime Stats

/status

Display runtime statistics like token count, throughput, etc.

🕰️ Show History

/history

Review the current session’s conversation history.

🔍 Toggle Verbose Mode

/verbose

Enable detailed performance metrics per turn. Run again to disable.

📦 List Models

Display all available models and locally downloaded models:

/list

👋 Quit CLI Mode

/bye

Exit the CLI.

🧠 Think Mode Toggle

Type /think to toggle Think Mode on or off interactively in the CLI.

💡 Note: This feature is only supported on certain models, such as Qwen3.

📂 Load a Local Text File in CLI Mode

Use any file that can be opened in Notepad (like .txt, .json, .csv, etc.).

Format (in CLI mode):

/input "<file_path>" prompt

Example:

/input "C:\Users\Public\Desktop\alice_in_wonderland.txt" Summarize it into 200 words

Notes:

• Use quotes only around the file path
• No quotes around the prompt
• File must be plain text (readable in Notepad)

👉 Download a sample prompt (around 40k tokens)

⚠️ Caution: a model’s supported context length is limited by available DRAM capacity. For example, with 32 GB of DRAM, LLaMA 3.1:8B cannot run beyond a 32K context length. For the full 128K context, we recommend a larger memory system.

If DRAM is heavily used by other programs while running FastFlowLM, you may encounter errors due to insufficient memory, such as:

[XRT] ERROR: Failed to submit the command to the hw queue (0xc01e0200):
Even after the video memory manager split the DMA buffer, the video memory manager
could not page-in all of the required allocations into video memory at the same time.
The device is unable to continue.

🤔 Interested in checking the DRAM usage?

1. Press Ctrl + Shift + Esc (or Ctrl + Alt + Del and select Task Manager).
2. Go to the Performance tab.
3. Click Memory to see total, used, and available DRAM, as well as usage percentage.

🌄 Loading Images in CLI Mode (for VLMs only, e.g. gemma3:4b)

Supports .png and .jpg formats.

/input "<image_path>" prompt

Example:

/input "C:\Users\Public\Desktop\cat.jpg" describe this image

Notes:

• Make sure the model you are using is a vision model (VLM) (e.g., gemma3:4b)
• Put quotes only around the file path
• Do not use quotes around the prompt
• Image must be in .jpg or .png format

⚙️ Set Variables

/set

Customize decoding parameters like top_k, top_p, temperature, context length (max), generate limit, etc.

⚠️ Note: Providing invalid or extreme hyperparameter values may cause inference errors. generate limit sets an upper limit on the number of tokens that can be generated for each response. Example:

/set gen-lim 128

📊 Benchmarking Tool

Use the FLM benchmarking tool to measure a model’s performance across different context lengths.

Each benchmark tests context lengths from 1k to 32k, running 2 iterations at each length.

flm bench llama3.2:1b

Change the iteration times by bench-iterations:

flm bench llama3.2:1b --bench-iterations 4

Cap the longest context tested with -c / --ctx-len:

flm bench llama3.2:1b --ctx-len 8192

The value is rounded down to a power-of-two number of k (e.g. -c 5000 tests 1k, 2k and 4k), and model-specific limits still apply. Without it, the benchmark runs up to 32k.

FLM prints the results in your terminal and also saves them as a CSV file in the current folder for later reference.

[FLM]  === Benchmark Results ===

 Context Length |              TTFT (s) |      Prefill Speed (tok/s) |     Decoding Speed (tok/s)
----------------------------------------------------------------------------------------------------
             1k |       0.815 ±   0.012 |        1233.57 ±     17.91 |       61.39 ±      0.46
             2k |       1.144 ±   0.035 |        1728.16 ±     55.68 |       59.04 ±      0.36
             4k |       1.955 ±   0.009 |        2001.55 ±      9.22 |       54.32 ±      0.47
             8k |       3.821 ±   0.010 |        2037.62 ±      5.40 |       46.89 ±      0.19
            16k |       9.144 ±   0.011 |        1698.20 ±      2.08 |       37.20 ±      0.14
            32k |      26.346 ±   0.037 |        1177.16 ±      1.53 |       26.38 ±      0.07
----------------------------------------------------------------------------------------------------

🗂️ Others

🛠 Change Default Context Length (max)

You can find more information about available models here:

C:\Program Files\flm\model_list.json

You can also change the default_context_length setting.

⚠️ Note: Be cautious! The system reserves DRAM space based on the context length you set.
Setting a longer default context length may cause errors on systems with smaller DRAM. Also, each model has its own context length limit (examples below).

• qwen3-tk:4b → up to 256k tokens
• gemma3:4b → up to 128k tokens
• gemma3:1b → up to 32k tokens
• llama3.x → up to 128k tokens