Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)
This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models.
Cloud Codes12 minTranscript found
Quick learning frame
Read this before watching.
AI-native interfaces are control surfaces for intent, artifacts, context, preview, inspection, and iteration.
New playlist item from Cloud Codes; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Skill you build: The ability to decide which weights of a sparse mixture-of-experts model deserve scarce VRAM and to tune llama.cpp offload flags and batch sizes accordingly, instead of treating VRAM capacity as a hard wall.
Watch for the shift from claim to mechanism. The learning value is the point where the transcript reveals a repeatable action, tool boundary, context move, review habit, or artifact.
Concept diagram
Where this video fits.
01Intent
02Context
03Generation surface
04Preview
05Critique
06Implementation handoff
Deep lesson
Turn this video into working knowledge.
2,028 cleaned transcript words reviewed across 596 timed caption segments.
Thesis
Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide) teaches a practical ai interface control move: This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models.
The goal is not to remember the video. The goal is to extract the operating principle, tie it to timestamped evidence, test how far the claim transfers, and make something reusable.
1:23
Sparsity beats capacity
“single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed experts are idle most of the time. Idle weights can live in system RAM. Until the middle of...”
Qwen3.6 35B A3B has 256 experts but the router wakes only eight plus one shared expert per token, so it moves under 1 GB of weights per token where a dense 35B at four bits would stream roughly 20 GB; that makes the real question not whether the model fits but which parts must be fast, so attention, embeddings, the shared expert, and the KV cache go to GPU while the 248 idle experts live in system RAM. Write out the per-token memory traffic for a dense 35B versus this 35B-A3B model at four bits, then list which tensors you would pin to VRAM and why.
6:54
Prompt versus decode
“pointing a coding agent at a repository, you will use every one of them. Here's what that costs on this specific model. 20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000...”
The CUDA backend compares every expert matmul against op_offload_min_batch_size, default 32: decoding one token at a time falls under it so the CPU does the math at about 51 GB/s (your 17 tok/s), while a pasted prompt of hundreds of tokens exceeds it, so llama.cpp ships expert weights over PCIe to the GPU; raising the 512-token micro-batch matters, and the maintainer's benchmark went from 22 to 345 tokens a second moving 128 up to 2048. Run one short chat turn and one long pasted-file prompt against the same local model and record decode versus prompt-processing throughput, then re-run with the micro-batch raised to 2048.
9:45
Where it breaks
“across the memory bus. That paper measured verification running two to three times slower and speculation making the whole system up to one and a half times slower than not speculating at all. The workaround is multi-token prediction,...”
TurboQuant's 3-bit rotated KV cache cuts KV memory at least six times and takes a 6 GB card from 64k to 256k context, but it is not in llama.cpp: all 75-odd related pull requests are closed unmerged (two tagged an AI policy violation), it needs a fork 300 commits ahead, and after the MoE attention kernel rewrite Turbo 2 decodes at 45 percent of F16. Speculative decoding fails too, because guessing several tokens ahead wakes most of the experts instead of eight, measured as up to 1.5 times slower than not speculating. List the three failure modes named here (bandwidth ceiling, unmerged TurboQuant, expert-waking speculation) and, for each, write the one measurement that would tell you it is hurting your setup.
01
Intent
Start with this video's job: This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models. Treat "Intent" as the outcome you are trying to make visible, not a topic label. Anchor it to 1:23, where the video says: “single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed experts are idle most of the time. Idle weights can live in system RAM. Until the middle of...”
02
Context
Use "Context" to locate the part of the ai interface control mechanism the video is demonstrating. Ask what changes in your real setup if this claim is true. Anchor it to 6:54, where the video says: “pointing a coding agent at a repository, you will use every one of them. Here's what that costs on this specific model. 20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000...”
03
Generation surface
Turn "Generation surface" into the reusable artifact for this lesson: A UI control-surface critique sheet with context inputs, artifact visibility, review criteria, and implementation handoff. This is where watching becomes something you can inspect and reuse.
04
Preview
Use "Preview" as the application surface. Decide whether the idea touches a browser flow, a local file, a model choice, a source document, a UI, or a review step.
05
Critique
Use "Critique" to prove the lesson. The evidence should connect back to the video title, transcript anchors, and a concrete output, not a generic best-practice claim.
06
Implementation handoff
Use "Implementation handoff" to carry the idea forward: save the prompt, checklist, diagram, or operating rule that would make the next agent run better.
Example
Source-backed artifact packet
Convert the video into a scoped artifact request that includes the transcript claim, mechanism, acceptance criteria, and proof. The output should be a ui control-surface critique sheet with context inputs, artifact visibility, review criteria, and implementation handoff..
Example
AI interface control proof brief
Separate what the speaker claims, what the demo actually proves, and what still needs outside verification before you adopt the ai interface control pattern.
Example
Teach-back module
Transform the lesson into a definition, a Intent -> Context -> Generation surface -> Preview -> Critique -> Implementation handoff diagram, one misconception, one practice exercise, and a check-for-understanding question.
Do not learn it wrong
Treating the title as the lesson without checking what the transcript actually says.
generic UI inspiration
visual output with no critique
handoff that lacks implementation criteria
Letting the lesson drift into generic design tips.
Letting the lesson drift into visual hype without inspection.
Letting the lesson drift into screenshots without implementation criteria.
Do not count this as learned until these are true.
01
State the transcript-backed claim in your own words: This video shows how a 22.36 GB four-bit Qwen3.6 35B A3B build runs at 17 tokens a second on a 6 GB GTX 1060 by using llama.cpp flags (--n-cpu-moe, --no-mmap, --n-gpu-layers, --mlock, and a larger micro-batch) to park idle mixture-of-experts weights in system RAM while attention, embeddings, the shared expert, and the KV cache stay on the GPU. It then names the three places the trick breaks: memory-bandwidth ceilings, TurboQuant KV compression that never merged upstream, and speculative decoding that backfires on sparse models.
02
Explain the practical stakes without hype: New playlist item from Cloud Codes; queued for transcript-backed review, topic mapping, and a practical learning artifact.
03
Map the idea onto the Intent -> Context -> Generation surface -> Preview -> Critique -> Implementation handoff sequence and name the weakest link.
04
Produce the artifact and include the evidence that proves it: A UI control-surface critique sheet with context inputs, artifact visibility, review criteria, and implementation handoff.
Put it into practice
Give this grounded prompt to Codex or Claude after watching.
You are helping me turn one specific YouTube video into real, durable learning.
Source video:
- Title: Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)
- URL: https://www.youtube.com/watch?v=7AExwNFlXU4
- Topic: Interfaces + Open Design
- My current learning frame: Load a four-bit mixture-of-experts model on your own GPU, walk --n-cpu-moe down from every layer until the loader runs out of VRAM, then push --n-gpu-layers back up and record tokens per second alongside your RAM speed.
- Why this matters: New playlist item from Cloud Codes; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Transcript anchors from this exact video:
- 1:23 / Evidence 1: "single token, so attention goes on the GPU. Embeddings, the shared expert, the key value cache, all GPU. The routed experts are idle most of the time. Idle weights can live in system RAM. Until the middle of..."
- 3:54 / Evidence 2: "of system memory, run faster with memory mapping left on because the page cache is doing real work. Flag three is arithmetic. Once the experts move out, most of your 6 gigs is empty again. So, push layers..."
- 6:54 / Evidence 3: "pointing a coding agent at a repository, you will use every one of them. Here's what that costs on this specific model. 20,480 bytes of key value cache per token per sequence, 20 KB. So, the full 256,000..."
- 9:45 / Evidence 4: "across the memory bus. That paper measured verification running two to three times slower and speculation making the whole system up to one and a half times slower than not speculating at all. The workaround is multi-token prediction,..."
- 11:18 / Evidence 5: "You have stopped buying capacity and started deciding which weights deserve the fast memory. 6 GB, a card from 2016, 35 billion parameters answering in real time. Go run it, then tell me your tokens per second and..."
Video-aware target:
- Prompt lane: AI interface control
- Mechanism to extract: Extract how the interface gives the user control over context, visual quality, generated artifacts, and handoff.
- Artifact to produce: A UI control-surface critique sheet with context inputs, artifact visibility, review criteria, and implementation handoff.
- Artifact must include: context input; visual target; preview/review step; implementation handoff; quality rubric
Your task:
1. Use the transcript anchors above as the primary source packet. If you add outside context, label it clearly as outside context and keep it secondary.
2. Create a source-check table with columns: timestamp, claim, transcript support, what the demo proves, confidence, and what still needs verification.
3. Extract the actual teachable mechanism from the video: Extract how the interface gives the user control over context, visual quality, generated artifacts, and handoff. Do not invent claims that are not supported by the title, lesson frame, or transcript anchors.
4. Build a reusable learning artifact: A UI control-surface critique sheet with context inputs, artifact visibility, review criteria, and implementation handoff.
5. Include:
- a plain-English definition of the core idea
- a diagram or structured model using this sequence: Intent -> Context -> Generation surface -> Preview -> Critique -> Implementation handoff
- answers to these source questions: What does the interface let the user control? | What artifact becomes visible? | What critique or handoff step closes the loop?
- 3 concrete examples that apply the video idea to real agentic work, such as design.md handoff; Figma-to-code review; UI reference library translation
- 2 failure modes the video helps prevent, chosen from the transcript evidence and these likely risks: generic UI inspiration; visual output with no critique; handoff that lacks implementation criteria
- a checklist for the next real workflow, focused on: context, preview, artifact visibility, critique, handoff
- one practical exercise with a clear done signal: Turn one UI demo into a design-review checklist for a real product screen.
6. Add a "learning transfer" section: what changes in my workflow tomorrow if I actually learned this?
7. Add a "source check" section that cites which transcript anchor supports each major takeaway.
Quality bar:
- Make this specific to "Running a 22GB AI Model on a 6GB GPU, FAST (llama.cpp Guide)", not a generic Interfaces + Open Design essay.
- Cite transcript anchors for every claim about design context, UI generation, preview, critique, or handoff.
- Prefer operational examples, failure modes, and reusable artifacts over broad definitions.
- Call out uncertainty instead of smoothing over weak evidence.
- Avoid these generic drifts: generic design tips; visual hype without inspection; screenshots without implementation criteria.
- If evidence is weak or missing, stop and say what transcript segment or timestamp needs review instead of guessing.
- Finish with a concise artifact I could paste into my learning app.
Misconceptions
What to stop believing.
A beautiful page is automatically a good learning tool.
Learning requires sequence, active recall, feedback, and application.
Generated UI should be accepted as-is.
Generated UI needs critique, revision, and browser verification.
Practice studio
Learning only counts when you make something.
01
Transcript evidence map
Separate what the video actually says from what you already believe about the topic.
3 source-backed takeaways with timestamps, confidence, and a transfer note.02
One useful artifact
Apply the video to a real workflow and produce a ui control-surface critique sheet with context inputs, artifact visibility, review criteria, and implementation handoff..
A reusable artifact with a done signal and one verification step.03
AI interface control teach-back card
Explain the ai interface control mechanism to someone who has not watched the video yet.
A 90-second explanation, one diagram, one example, and one misconception to avoid.
Recall check
Answer first, then reveal — without rewatching.
Why can a 22 GB model run usefully on a 6 GB card when a dense model of the same parameter count cannot?
What decides whether an expert matrix multiply runs on the CPU or gets copied to the GPU?
Why does the video say TurboQuant KV compression is a memory win rather than a speed win, and why is it hard to use?
Source shelf
Use the video as a doorway, then verify with primary sources.
