Qwen3.8 Flash Next UD-Q4_K_XL 49 tokens/s TGS using 2x RTX 3090 on Windows 11.

Developer's own thread: https://www.reddit.com/r/LocalLLaMA/s/adp1cGZZe9

Code: https://github.com/Inovello/llama.cpp/tree/flashnext-e06

My hardware: 2x RTX 3090, Intel Ultra 7 270k Plus, 192 GB DDR5@5600 MHz

Token generation speed went from 20 t/s to 49 t/s. Prompt processing speed is 140 t/s. Prompt processing is faster on the main branch.

https://preview.redd.it/an5rtqz5nwoh1.png?width=643&format=png&auto=webp&s=6b9f8760d2440e4a40420f27956179808826017a

I have CUDA 13.3.1 installed. I use Windows 11 and I used these commands to compile it with the latest Visual Studio:

$vcvars = "C:\Program Files\Microsoft Visual Studio\18\Community\VC\Auxiliary\Build\vcvars64.bat" cmd /c "`"$vcvars`" >nul 2>&1 && set" | ForEach-Object { if ($_ -match '^([^=]+)=(.*)$') { [System.Environment]::SetEnvironmentVariable($matches[1], $matches[2]) } } cmake -G Ninja -B build -S . -DCMAKE_BUILD_TYPE=Release ` -DGGML_CUDA=ON -DGGML_CCACHE=OFF -DGGML_NATIVE=ON cmake --build build --target llama-cli llama-bench llama-server -j 

My bat script:

@echo off echo Using CUDA backend with 2x RTX 3090s set LLAMA_ATTN_ROT_DISABLE=1 set LLAMA_MMAP_PIN_HOST=1 "C:\Users\server\Desktop\llama.cpp-flashnext-e06\build\bin\llama-server.exe" ^ --host 0.0.0.0 ^ --port 8081 ^ --alias Qwen3.8-Flash-Next ^ --model H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf ^ --temp 1.0 ^ --top-p 0.95 ^ --top-k 20 ^ --min-p 0.0 ^ --presence-penalty 0.0 ^ --repeat-penalty 1.0 ^ --ctx-size 262144 ^ -ot "ffn_(gate|up|down)_exps\.weight=CUDA_Host,per_layer_token_embd\.weight=CPU" ^ --moe-expert-cache 150 ^ --ubatch-size 512 ^ --batch-size 4096 ^ --mmproj H:\Qwen3.8-Flash-Next\mmproj-Qwen3.8-Flash-Next-BF16.gguf ^ --no-mmproj-offload ^ --threads 22 ^ --threads-batch 22 ^ --spec-type draft-mtp,ngram-mod ^ --model-draft H:\Qwen3.8-Flash-Next\UD-Q4_K_XL\MTP\mtp-Qwen3.8-Flash-Next-shared-Q8_0.gguf ^ --spec-draft-n-max 2 ^ --spec-ngram-mod-n-match 60 ^ --spec-ngram-mod-n-min 12 ^ --spec-ngram-mod-n-max 24 ^ --flash-attn on ^ --kv-offload ^ --cache-type-k bf16 ^ --cache-type-v bf16 ^ --parallel 1 ^ --jinja ^ --reasoning-preserve ^ --chat-template-kwargs "{\"reasoning_effort\":\"xhigh\"}" ^ --no-warmup ^ --load-mode none ^ --lazy-mode off pause