Hot Expert Reload on GPU is what this community needs
r/LocalLLaMA
September 11, 2026 at 12:25 PM
📌 A huge favor to ask llama maintainers - please implement this feature. Even with one 3090 card there will be tangible improvements in decode speed on MOE models with moderate number of active parameters, like Qwen3.8-Flash-Next, Deepseek V4/V4.1 Flash, GLM 5.3 Flash. With 2x 3090 speeds will be quite close to the full offload of these models to VRAM. This will make these almost SOTA models really usable locally. submitted by /u/perelmanych [link] [comments]
A huge favor to ask llama maintainers - please implement this feature. Even with one 3090 card there will be tangible improvements in decode speed on MOE models with moderate number of active parameters, like Qwen3.8-Flash-Next, Deepseek V4/V4.1 Flash, GLM 5.3 Flash. With 2x 3090 speeds will be quite close to the full offload of these models to VRAM. This will make these almost SOTA models really usable locally.
[link] [comments]
Read the full article at
r/LocalLLaMA
More in Models & Releases
Models & Releases
Nvidia's RTX 5090 vanishes from online retail in the US — third-party sellers now demand as much as $9,500 for Nvidia's fastest GPU
submitted by /u/Norwood_Reaper_ [link] [comments]
Jensen Huang puts Trump on speakerphone onstage to announce robots won’t take over the world
Nvidia CEO Jensen Huang took a call from President Trump on Monday while onstage at the All-In Podcast's All-In Summit. It's not the first time Huang has taken a call from the president during work, but this time he put Trump on speakerp...
Models & Releases
Base-10's Charlie O'Neill on why Kimi and GLM are "almost objectively" better than Opus 5
Edit: Spelled Baseten not Base-10 Full episode of this available at https://www.youtube.com/watch?v=PrSf7IOYu-I It's interesting to see how Dwarkesh has had to come around to the evidence that we are well on our way to creating AGI and e...
Models & Releases
NVIDIA Unveils RTX PRO 5500 "Blackwell" Workstation GPU with 84 GB GDDR7 Memory
submitted by /u/Lumpy_Phase_9539 [link] [comments]