local-ai-coding-setup
Local AI coding on a 6 GB GPU
The free, fully local AI coding setup I run every day on my laptop: a coding agent in the terminal plus autocomplete in VS Code, with no subscription and no cloud model. It runs on your own PC, so there's nothing to host and no demo to open. This page is the short version, and the repo has everything else.
01 what you get
What you get
The cleaned-up public version of the setup: scripts, configs and measurements. The models (about 17 GiB, 28 with the optional Qwen3.6) live outside the repo, in C:\AI by default.
-
agent
A coding agent in the terminal
Pi talks to
llama-serveron:8080(OpenAI-compatible API). I always start it throughscripts/pi-local.ps1, which hides cloud API keys from the session and turns off telemetry and version checks. -
autocomplete
Autocomplete in VS Code
A second
llama-serveron:8012serves fill-in-the-middle autocomplete to the llama-vscode extension. It runs on the CPU by default, so the GPU stays with the main model. -
safety
A permission gate
An extension asks before risky commands (
git push,ssh,rm -rf,migrate:fresh, writes outside the project) and blocks them in non-interactive runs. Its tests are 254 checks that need no model. -
models
Presets that fit
Gemma-4-26B-A4B is the default agent and Qwen3.5-9B is fast mode, fully on the GPU. Ollama on
:11434serves the same 9B forai-pi -Fast, and Qwen3.6-35B-A3B is an optional download. -
scripts
Start and stop on demand
ai-on,ai-fast,ai-pi,ai-statusandai-offwrap the scripts.-DryRunon any start script prints the command without starting anything. Nothing autostarts, because the laptop is also my gaming machine. -
measurements
Real numbers
Benchmarks from this laptop, plus an A/B test that ran each model through Pi on five tasks in a clone of a real Laravel app, with hidden tests deciding the result.
- pi in terminal
- permission gate
- llama-server
- gpu + ram split
- llama.cpp
- Ollama
- Pi
- llama-vscode
- PowerShell
02 hardware
The hardware it targets
Everything was measured on the laptop the setup runs on. The GPU has 6 GB, of which about 5.1 GB is usable for models, and the agent models are 10-11 GiB files.
- laptop
- ASUS TUF A15 (FA506QM)
- cpu
- Ryzen 9 5900HX
- gpu
- RTX 3060 Laptop 6 GB
- ram
- 16 GB
- os
- Windows 11
-
-ncmoe
How a 10 GiB model fits
The big models are mixture-of-experts.
-ncmoe Nkeeps the expert weights of the first N layers in system RAM and the rest on the GPU, which is how a 10 GiB model runs next to 6 GB of VRAM. -
sysmem fallback off
No quiet spill
The NVIDIA driver is set to "Prefer No Sysmem Fallback" (
scripts/nvidia-sysmem.py), so a config that doesn't fit fails to load instead of quietly spilling into system RAM. -
16 GB
RAM is the real limit
RAM is the real limit, not VRAM. With the MoE models loaded through mmap, free RAM drops below 300 MB (67 MB at worst).
03 numbers
Models and numbers
Measured with llama.cpp b11429 (CUDA 12.4), 64K context (32K for the 9B), q8_0 KV cache, flash attention, 12 threads, and a prompt of about 8K tokens of real PHP code. The agent rows had autocomplete on the CPU.
| Role | Model | Fit | Generation | Cold 8K prompt | VRAM peak |
|---|---|---|---|---|---|
| Agent (default) | Gemma-4-26B-A4B-it, UD-IQ3_S, 10.5 GiB | experts of 24 layers on CPU | 27-28 tok/s | 282 tok/s | 5265 MiB |
| Agent (kept for comparison) | Qwen3.6-35B-A3B, UD-IQ2_M, 10.7 GiB | experts of 28 layers on CPU, -lm none | 36.5-38.7 tok/s | 390 tok/s | 5427 MiB |
| Fast mode | Qwen3.5-9B, IQ4_XS, 4.8 GiB | fully on GPU | ~55 tok/s (~57 via Ollama) | ~1000-1500 tok/s | 5321 MiB |
| Autocomplete | Qwen2.5-Coder 0.5B Q8_0 (CPU) / 1.5B Q8_0 (GPU) | 0.45-0.52 s / 0.12-0.22 s per request | 0 / 2053 MiB |
The Qwen3.6 row was measured with Chrome open, the Gemma row with it closed.
Speed alone didn't pick the default
I ran each model through Pi on five tasks in a clone of a real Laravel app (1053 tests green at baseline). A task passes when hidden tests pass (a 10-point rubric for the explain task). The score is a blind 1-5 review of the diff, given without knowing which model wrote it.
| Model | Passed | Median time, passed tasks | Review score (tasks 1-4) |
|---|---|---|---|
| Gemma-4-26B-A4B | 4/5 | 255 s | 4.0 |
| Qwen3.5-9B | 4/5 | 65 s | 3.0 |
| Qwen3.6-35B-A3B | 3/5 | 142 s | 3.7 (3 tasks) |
| gpt-oss-20b | 0/4 | - | - |
No model produced a broken tool call. Gemma and the 9B tied on passes; Gemma won on review score and had no hidden bugs in production code, so it became the default.
04 run it
Quick start
Windows 11, PowerShell 7 or 5.1, an NVIDIA GPU, Node 24, Python with uv. Versions I use: llama.cpp b11429, Ollama 0.40.1, Pi 1.1.0, llama-vscode 0.0.68. Models and llama.cpp go under C:\AI.
-
Clone the repo. The later steps run from inside it.
git clone https://github.com/ramaraihan12/local-ai-coding-setup cd local-ai-coding-setup -
Unzip
llama-b11429-bin-win-cuda-12.4-x64.zipandcudart-llama-bin-win-cuda-12.4-x64.zipfrom the llama.cpp release tagb11429intoC:\AI\llama.cpp. -
Install the
hfCLI (uv tool install huggingface_hub) and download the models (checksums).$G = "C:\AI\models\gguf" hf download unsloth/gemma-4-26B-A4B-it-GGUF --include "gemma-4-26B-A4B-it-UD-IQ3_S.gguf" --local-dir "$G\unsloth\gemma-4-26B-A4B-it-GGUF" hf download unsloth/Qwen3.5-9B-GGUF --include "Qwen3.5-9B-IQ4_XS.gguf" --local-dir "$G\unsloth\Qwen3.5-9B-GGUF" hf download ggml-org/Qwen2.5-Coder-0.5B-Q8_0-GGUF --include "qwen2.5-coder-0.5b-q8_0.gguf" --local-dir "$G\ggml-org\Qwen2.5-Coder-0.5B-Q8_0-GGUF" hf download ggml-org/Qwen2.5-Coder-1.5B-Q8_0-GGUF --include "qwen2.5-coder-1.5b-q8_0.gguf" --local-dir "$G\ggml-org\Qwen2.5-Coder-1.5B-Q8_0-GGUF" # optional, only for -Preset qwen36 hf download unsloth/Qwen3.6-35B-A3B-GGUF --include "Qwen3.6-35B-A3B-UD-IQ2_M.gguf" --local-dir "$G\unsloth\Qwen3.6-35B-A3B-GGUF" -
Install Ollama (
winget install --id Ollama.Ollama -e), turn off its autostart, setOLLAMA_MODELS=C:\AI\models\ollama, thenollama create qwen3.5-9b:iq4xs -f configs\ollama\Modelfile.qwen35-9b. -
Optional: add
llama-server.exeandllama-bench.exein NVIDIA Control Panel > Program Settings once, thenpython -I scripts\nvidia-sysmem.py --apply(--checkonly reads). -
Pi:
npm i -g @earendil-works/pi-coding-agent@1.1.0,winget install BurntSushi.ripgrep.MSVC sharkdp.fd, thenpi install git:github.com/obra/superpowers@v7.0.0and copy the config.$A = "$env:USERPROFILE\.pi\agent"; New-Item -ItemType Directory -Force $A | Out-Null Copy-Item configs\pi\models.json, configs\pi\settings.json, configs\pi\mcp.json $A Copy-Item configs\pi\global-rules.md "$A\AGENTS.md" Copy-Item configs\pi\extensions, configs\pi\prompts, configs\pi\mcp-servers $A -Recurse -Force uv tool install "duckduckgo-mcp-server[browser]==0.7.0" --with "ddgs==9.16.0" -
VS Code: install llama-vscode and copy
configs/vscode/settings.jsoninto your user settings. -
Add one line to
$PROFILE(works in pwsh 7 and 5.1).$s = '<clone>\scripts\ai-shortcuts.ps1'; if (Test-Path -LiteralPath $s) { . $s }
Daily use
ai-on # autocomplete on CPU (:8012) + Gemma (:8080), waits until healthy
ai-fast # same, with Qwen3.5-9B fully on the GPU instead
ai-pi # Pi in the current folder; ai-pi -Lite skips skills and MCP for a smaller prompt
ai-status # servers, loaded models, VRAM, RAM
ai-off # stop everything; VRAM should read 0 MiB
05 limits
What to keep in mind
- "Local" means the models. Context7 docs and web search in Pi go over the network (
-Liteturns them off). - The gate isn't a sandbox. The permission gate is an emergency brake against prompt injection.
- Windows only. It depends on NVAPI, the Ollama tray app and Windows paths in the gate.
- The 9B needs review. It's about 4x faster per task, but it left a bug that passed the tests. I use it for small things and review everything it writes.
- Thinking can eat the budget. It can use the whole 8192-token output budget before any edit. Qwen3.6 failed an A/B task that way.
docs
The long version
- Why this stack llama.cpp, Ollama, MoE models at 2-3 bits, Pi and the permission gate
- Presets and the 6 GB fit the
-ncmoesweep, sysmem fallback, files and checksums - Benchmarks llama-bench sweeps, server runs, RAM and thermals
- A/B test local models on real coding tasks, with method and per-task data