DeepSeek V4.1 Flash on 8× A40: ~40 tok/s Q2_K and ~32 tok/s Q4_K_M with TensorSharp

Hey everyone — I’m building TensorSharp, an open-source LLM inference engine. Here are the latest DeepSeek V4.1 Flash GGUF results using its native ggml_cuda backend.

Setup: 8× NVIDIA A40, layer split, F16 KV cache, 65,536-token configured context. Prefill measurements use approximately 4.9K-token prompts—not the full context window.

Final optimized results — all speeds in tokens/sec:

Metric |Q2_K |Q4_K_M
Prefill |533–539 |451.8–492.1
Single-request decode |40.31–40.72 |31.0–32.5
Decode, 2 concurrent requests — total |— |39.3
Decode, 4 concurrent requests — total |— |48.9
Decode, 8 concurrent requests — total |— |48.5 — = not reported in the cited final Q2_K retest. Concurrent figures are aggregate throughput, not per-request speed.

What made the difference:

• Q2_K: Keeping the ~60 GiB quantized Engram tables on GPUs removes storage-dependent lookups. Unifying the backend per GPU also cuts decode graph splits from roughly 570 to 8, reducing synchronization overhead.
• Q4_K_M: The larger Engram tables stay in host memory. Automatic warming, tighter VRAM budgeting and token-batched decode deliver ~1.9× prefill and ~2× four-request throughput versus the previous behavior. Only 1 of 40 layers keeps its routed experts on the CPU.
• More parallelism isn’t automatically faster: On this no-NVLink system, Q4_K_M layer split beats experimental routed-MoE tensor parallelism: 31–32.5 vs. 21.4–22 tok/s single-stream decode.

These are project-reported throughput measurements, not cold-start timings or a head-to-head win over another engine. Numerical/model-quality parity remains unproven, and batching can change generated output.