Search

Make your llama generation time fly with AWS Inferentia2

Comparing the Performance of LLMs: A Deep Dive into Roberta, Llama 2, and Mistral for Disaster Tweets Analysis with Lora

Non-engineers guide: Train a LLaMA 2 chatbot

Llama 2 on Amazon SageMaker a Benchmark

Fine-tuning Llama 2 70B using PyTorch FSDP

Fine-tune Llama 2 with DPO

Llama 2 is here - get it on Hugging Face

StackLLaMA: A hands-on guide to train LLaMA with RLHF

Spomin - Live KV cache compaction (Experimental for Qwen)

I’ve been building "Spomin", a router that replaces context with summaries directly in the KV cache. The goal is to keep long running sessions going without repeatedly stopping for full compaction and reprocessing the context that remain...

Hot Expert Reload on GPU is what this community needs

A huge favor to ask llama maintainers - please implement this feature. Even with one 3090 card there will be tangible improvements in decode speed on MOE models with moderate number of active parameters, like Qwen3.8-Flash-Next, Deepseek...

llama.cpp ngram on RAM/SSD?

I've been out of the loop for some time. Is there already an official way to offload ngram to RAM or SSD in something like Unsloth Studio? Interested in running Qwen3.8-Flash-Next on 72GB VRAM, but naiive attempts failed because even at ...

Got an old slow low vram GPU laying around? Might be worth it to use for Just Vision mmproj llama.cpp

For many, Vram is precious, I see many people recommend using --no-mmproj-offload to save gpu vram but it is painfully slow. Especially if you are using it with agentic coding. If possible, add that secondary gpu just for mmproj with --m...

"Matrix Bros", a mash-up of Matrix and Super Mario vibe coded using Ornith-1.5-35B-Q4_K_M with 6GB VRAM with (almost) a single prompt :-)

I've seen a lot of Super Mario showcases, but never a mashup or derivative work. So I took a Super Mario generation prompt from ChopSticksPlease's Post here - and added a Matrix twist - and found myself super satisfied with the results u...

The Local LLM community feels like the golden era of the internet all over again

Lately because of the current hardware shortage, unfortunately or fortunately, we can’t just throw infinite cloud compute at our problems, but we’re forced to actually care about what’s happening under the hood. We’re tweaking inference ...

llm performance community metric

my question about LLM performance We see a lot of posts about token prediction, token generation per second, etc. But is it really the metric? I can see that DeepSeek V4 Flash 0731 (with DSPark; mac studio + llama.cpp) produces about 22–...

Qwen3.8 flash next - untrained svg generation

> "make an svg of a frog playing on a chello on the back of a whale with carribean island in the back." interestingly the svg looks different in the OpenWebUi preview then when looked at in preview (osx). The palms and music notes are mi...

Qwen 3.8 27B UD-IQ4_XS even faster on 16GB CUDA

This is an evolution on top of Raymond's KV cache streaming fork - all credits to what enabled this goes to him. The basic idea behind what he enabled was a pool of memory in VRAM that is used differently depending on the phase (prompt p...

Dear 24G owners, try VLLM you might be able to run Qwen3.8 27B INT4, 144K FP8 KV on RTX 3090 with better speed. (TLDR VLLM AOT)

VLLM Benchmark: Prefill, Prompt processing - avg, 871.93 tok/s (3 hours constant running xhigh) - 10K prompt, 1000.26 tok/s (16 runs) - 90K prompt, 743,59 tok/s (16 runs) Decode, tok gen - avg, 38.39 tok/s (3 hours constant running xhigh...

I have just moved from MacBook M5 pro 48 GB to RTX3090

Using the RTX3090 on a linux machine I built for it and running Qwen 3.8 27B getting average 100t/s compared to my MacBook 20t/s I think I can finally get rid of my Claude subscription, this is good enough for me. I am a software dev and...

Micron's memory wall chart. Compute up ~3x every two years, HBM bandwidth under 2x

From Raghu Sreeramaneni's memory tutorial at hot chips 2026. top line is normalised tflops for tpu v3 through r200, bottom line is hbm2e through hbm4, both log scale, so the distance between them is a lot wider than it looks. The three b...