Search

NVIDIA PAIR routing to llama.cpp on an AMD ROCm node (2×R9700). Notes.

PAIR (NVIDIA's Personal AI Router) ships GPU telemetry only for NVIDIA nodes — AMD nodes report a blind fallback, so the scheduler can't see their load. I added AMD ROCm telemetry and got a two-node cluster routing to my tuned llama.cpp ...

Data point: Qwen3.8-Flash-Next PP/TG speed on M3 Ultra

Model File: https://huggingface.co/unsloth/Qwen3.8-Flash-Next-GGUF Q4_K_XL llama.cpp configuration through llama-swap: -c 256000 --jinja --temp 1 --top-p 0.95 --top-k 20 --min-p 0.00 --presence-penalty 0 --repeat-penalty 1.0 Testing by:...

What pi.dev plugin do you suggest for context, compaction and memory management of local models?

I have been battling with my Qwen3.8:27b setup on my rtx 5080 16gb. I am using llama.cpp to run a nvfp4 version of qwen3.8:27b llama-b10699-bin-win-cuda-13.3-x64\llama-server.exe -hf williamliao/Qwen3.8-27B-NVFP4-GGUF:NVFP4 --jinja --cha...

CodeFinetuner: Fine-tune a local code autocomplete model on your own codebase

Hi everyone, I was interested in learning LoRA fine-tuning, and ended up building CodeFinetuner over the past few months, a full pipeline that fine-tunes a small code autocomplete model (e.g. Qwen2.5-Coder-3B) specific to a codebase. You...

Qwen3.8 Flash Next llama.cpp config tuning

Hola all. Do you guys mind sharing your LLama.cpp config and system setup details for Qwen3.8 Flash Next? Model's quite big and tryining many combinations of llama.cpp options takes lots of time, so looking for other people setup detail...

I built a local way to export, search and continue chats from OpenRouter, LM Studio and AI Studio with llama.cpp or OpenRouter

I had a lot of chats in OpenRouter across different models, with basically no proper way to bulk export or search them. So I wrote a js scraper for that. LM Studio was easier because chats are local, while Google AI Studio had the uglies...

Llama can now see and run on your device - welcome Llama 3.2

Code Llama: Llama 2 learns to code

Qwen3.8 Flash Next now at 1.2k t/s prefill on Strix Halo

As you all know, Qwen3.8 Flash Next on mainline llama.cpp is still in a pretty experimental stage, but a lot of community forks are trying to get it to work better. There's also a closed-source solution called Halogen (https://github.com...

Is there still strong interest in a dense 9b model?

I have a full model, it's ready to train. It's ~9b parameters. 9.4b to be more exact. That includes a 1/2/3 Engram table, Moonshot's AttnRes modeling, and RoPE / NoPE layering at 3:1 as more or less validated by most major labs. It uses ...

GGML and llama.cpp join HF to ensure the long-term progress of Local AI

New in llama.cpp: Model Management

Measuring Open-Source Llama Nemotron Models on DeepResearch Bench

Welcome the NVIDIA Llama Nemotron Nano VLM to Hugging Face Hub

Welcoming Llama Guard 4 on Hugging Face Hub

Welcome Llama 4 Maverick & Scout on Hugging Face

“Llama 3.2 in Keras”

Deploy Meta Llama 3.1 405B on Google Cloud Vertex AI

Llama 3.1 - 405B, 70B & 8B with multilinguality and long context

Welcome Llama 3 - Meta's new open LLM