Uploaded August 2026 | Updated September 2026, 2 weeks ago
I ran Qwen 3.8, Meta's Muse Glimmer and Gemma 4 through the same 12 accounting questions, 10 times each, on one RTX 5090, to find out whether a local LLM actually gives you the same answer twice.
Qwen 3.8 27B vs Muse Glimmer 30B vs Gemma 4 26B Consistency Test
kgptalkie.com/tutorials/generative-ai/qwen-3-8-27b-vs-muse-glimmer-vs-gemma-4-consistency
This is a consistency test, not a coding test. Every one of these models is probabilistic, so the same prompt can produce a different answer on every run, and that matters if you are putting a local model anywhere near real numbers. I share the exact setup so you can reproduce the runs, explain where the speed comes from (MTP, drafting and DFlash), then go through the consistency and accuracy numbers question by question. The result that surprised me most: one model was 100% consistent and 0% accurate at the same time.
📺 Full playlist: youtube.com/playlist?list=PLc2rvfiptPSReropGbvDFpB6dneNBwqhD
⏱ Chapters:
0:00 What a consistency test actually measures
2:18 My exact setup: RTX 5090, Ollama, Q4_K_M, 8K context
3:18 The benchmark: 12 accounts payable questions, 10 runs each
5:24 Where the speed comes from: MTP explained
7:10 Drafting on and off in Ollama
7:43 Why Muse Glimmer needs DFlash instead
8:58 Gemma 4 26B is MoE with only 3B active
10:46 The RTX 5090 ceiling: 107 tokens per second
12:16 DFlash speed: up to 252 tokens per second
14:22 Reasoning tokens and why benchmarks disagree
17:31 How decoding works: greedy vs top-k and top-p
20:37 Consistency results with the greedy sampler
21:52 Default sampler: Qwen 98.3%, Gemma 4 93%
22:38 Consistent does not mean accurate
25:25 Where Gemma 4 failed: problem 9
27:03 On llama.cpp: 173 vs 278 tokens per second
27:22 The DFlash trade-off: 275 tok/s at 75% accuracy
29:27 Final takeaway
🔗 Models and tools used:
Qwen 3.8 27B on Ollama: ollama.com/library/qwen3.8
Gemma 4 26B on Ollama: ollama.com/library/gemma4
Meta Muse Glimmer 30B: huggingface.co/meta-models/Muse-Glimmer-30B
Muse Glimmer GGUF and DFlash drafter: huggingface.co/meta-models/Muse-Glimmer-30B-GGUF
llama.cpp: github.com/ggml-org/llama.cpp
Setup: RTX 5090 32GB, Q4_K_M quantization, 8K context, 12 questions x 10 runs per model
📺 Watch next:
Qwen 3.8 27B Speed Settings Explained: MTP, KV Cache and Flash Attention: youtube.com/watch?v=QkzEkfIzvBk
🎓 Go deeper with my Udemy course:
Master Langchain v1 and Ollama - Chatbot, RAG and AI Agents: kgptalkie.com/langchain
If this was useful, hit like, and tell me in the comments which problem statement you want me to benchmark next, because I do read all of them and I am happy to run it. Subscribe and turn on the bell if you want the rest of the local LLM benchmarks.
#Qwen3 #LocalLLM #Ollama #RTX5090
I ran Qwen 3.8, Meta's Muse Glimmer and Gemma 4 through the same 12 accounting questions, 10 times each, on one RTX 5090, to find out whether a local LLM actually gives you the same answer twice.
Qwen 3.8 27B vs Muse Glimmer 30B vs Gemma 4 26B Consistency Test
kgptalkie.com/tutorials/generative-ai/qwen-3-8-27b-vs-muse-glimmer-vs-gemma-4-consistency
This is a consistency test, not a coding test. Every one of these models is probabilistic, so the same prompt can produce a different answer on every run, and that matters if you are putting a local model anywhere near real numbers. I share the exact setup so you can reproduce the runs, explain where the speed comes from (MTP, drafting and DFlash), then go through the consistency and accuracy numbers question by question. The result that surprised me most: one model was 100% consistent and 0% accurate at the same time.
📺 Full playlist: youtube.com/playlist?list=PLc2rvfiptPSReropGbvDFpB6dneNBwqhD
⏱ Chapters:
0:00 What a consistency test actually measures
2:18 My exact setup: RTX 5090, Ollama, Q4_K_M, 8K context
3:18 The benchmark: 12 accounts payable questions, 10 runs each
5:24 Where the speed comes from: MTP explained
7:10 Drafting on and off in Ollama
7:43 Why Muse Glimmer needs DFlash instead
8:58 Gemma 4 26B is MoE with only 3B active
10:46 The RTX 5090 ceiling: 107 tokens per second
12:16 DFlash speed: up to 252 tokens per second
14:22 Reasoning tokens and why benchmarks disagree
17:31 How decoding works: greedy vs top-k and top-p
20:37 Consistency results with the greedy sampler
21:52 Default sampler: Qwen 98.3%, Gemma 4 93%
22:38 Consistent does not mean accurate
25:25 Where Gemma 4 failed: problem 9
27:03 On llama.cpp: 173 vs 278 tokens per second
27:22 The DFlash trade-off: 275 tok/s at 75% accuracy
29:27 Final takeaway
🔗 Models and tools used:
Qwen 3.8 27B on Ollama: ollama.com/library/qwen3.8
Gemma 4 26B on Ollama: ollama.com/library/gemma4
Meta Muse Glimmer 30B: huggingface.co/meta-models/Muse-Glimmer-30B
Muse Glimmer GGUF and DFlash drafter: huggingface.co/meta-models/Muse-Glimmer-30B-GGUF
llama.cpp: github.com/ggml-org/llama.cpp
Setup: RTX 5090 32GB, Q4_K_M quantization, 8K context, 12 questions x 10 runs per model
📺 Watch next:
Qwen 3.8 27B Speed Settings Explained: MTP, KV Cache and Flash Attention: youtube.com/watch?v=QkzEkfIzvBk
🎓 Go deeper with my Udemy course:
Master Langchain v1 and Ollama - Chatbot, RAG and AI Agents: kgptalkie.com/langchain
If this was useful, hit like, and tell me in the comments which problem statement you want me to benchmark next, because I do read all of them and I am happy to run it. Subscribe and turn on the bell if you want the rest of the local LLM benchmarks.
#Qwen3 #LocalLLM #Ollama #RTX5090
