local-ai-coding-setup

Local AI coding on a 6 GB GPU

The free, fully local AI coding setup I run every day on my laptop: a coding agent in the terminal plus autocomplete in VS Code, with no subscription and no cloud model. It runs on your own PC, so there's nothing to host and no demo to open. This page is the short version, and the repo has everything else.

View on GitHub Quick start

01 what you get

What you get

The cleaned-up public version of the setup: scripts, configs and measurements. The models (about 17 GiB, 28 with the optional Qwen3.6) live outside the repo, in C:\AI by default.

  • agent

    A coding agent in the terminal

    Pi talks to llama-server on :8080 (OpenAI-compatible API). I always start it through scripts/pi-local.ps1, which hides cloud API keys from the session and turns off telemetry and version checks.

  • autocomplete

    Autocomplete in VS Code

    A second llama-server on :8012 serves fill-in-the-middle autocomplete to the llama-vscode extension. It runs on the CPU by default, so the GPU stays with the main model.

  • safety

    A permission gate

    An extension asks before risky commands (git push, ssh, rm -rf, migrate:fresh, writes outside the project) and blocks them in non-interactive runs. Its tests are 254 checks that need no model.

  • models

    Presets that fit

    Gemma-4-26B-A4B is the default agent and Qwen3.5-9B is fast mode, fully on the GPU. Ollama on :11434 serves the same 9B for ai-pi -Fast, and Qwen3.6-35B-A3B is an optional download.

  • scripts

    Start and stop on demand

    ai-on, ai-fast, ai-pi, ai-status and ai-off wrap the scripts. -DryRun on any start script prints the command without starting anything. Nothing autostarts, because the laptop is also my gaming machine.

  • measurements

    Real numbers

    Benchmarks from this laptop, plus an A/B test that ran each model through Pi on five tasks in a clone of a real Laravel app, with hidden tests deciding the result.

  1. pi in terminal
  2. permission gate
  3. llama-server
  4. gpu + ram split
  • llama.cpp
  • Ollama
  • Pi
  • llama-vscode
  • PowerShell
Read the code

02 hardware

The hardware it targets

Everything was measured on the laptop the setup runs on. The GPU has 6 GB, of which about 5.1 GB is usable for models, and the agent models are 10-11 GiB files.

laptop
ASUS TUF A15 (FA506QM)
cpu
Ryzen 9 5900HX
gpu
RTX 3060 Laptop 6 GB
ram
16 GB
os
Windows 11
  • -ncmoe

    How a 10 GiB model fits

    The big models are mixture-of-experts. -ncmoe N keeps the expert weights of the first N layers in system RAM and the rest on the GPU, which is how a 10 GiB model runs next to 6 GB of VRAM.

  • sysmem fallback off

    No quiet spill

    The NVIDIA driver is set to "Prefer No Sysmem Fallback" (scripts/nvidia-sysmem.py), so a config that doesn't fit fails to load instead of quietly spilling into system RAM.

  • 16 GB

    RAM is the real limit

    RAM is the real limit, not VRAM. With the MoE models loaded through mmap, free RAM drops below 300 MB (67 MB at worst).

03 numbers

Models and numbers

Measured with llama.cpp b11429 (CUDA 12.4), 64K context (32K for the 9B), q8_0 KV cache, flash attention, 12 threads, and a prompt of about 8K tokens of real PHP code. The agent rows had autocomplete on the CPU.

RoleModelFitGenerationCold 8K promptVRAM peak
Agent (default)Gemma-4-26B-A4B-it, UD-IQ3_S, 10.5 GiBexperts of 24 layers on CPU27-28 tok/s282 tok/s5265 MiB
Agent (kept for comparison)Qwen3.6-35B-A3B, UD-IQ2_M, 10.7 GiBexperts of 28 layers on CPU, -lm none36.5-38.7 tok/s390 tok/s5427 MiB
Fast modeQwen3.5-9B, IQ4_XS, 4.8 GiBfully on GPU~55 tok/s (~57 via Ollama)~1000-1500 tok/s5321 MiB
AutocompleteQwen2.5-Coder 0.5B Q8_0 (CPU) / 1.5B Q8_0 (GPU)0.45-0.52 s / 0.12-0.22 s per request0 / 2053 MiB

The Qwen3.6 row was measured with Chrome open, the Gemma row with it closed.

Speed alone didn't pick the default

I ran each model through Pi on five tasks in a clone of a real Laravel app (1053 tests green at baseline). A task passes when hidden tests pass (a 10-point rubric for the explain task). The score is a blind 1-5 review of the diff, given without knowing which model wrote it.

ModelPassedMedian time, passed tasksReview score (tasks 1-4)
Gemma-4-26B-A4B4/5255 s4.0
Qwen3.5-9B4/565 s3.0
Qwen3.6-35B-A3B3/5142 s3.7 (3 tasks)
gpt-oss-20b0/4--

No model produced a broken tool call. Gemma and the 9B tied on passes; Gemma won on review score and had no hidden bugs in production code, so it became the default.

04 run it

Quick start

Windows 11, PowerShell 7 or 5.1, an NVIDIA GPU, Node 24, Python with uv. Versions I use: llama.cpp b11429, Ollama 0.40.1, Pi 1.1.0, llama-vscode 0.0.68. Models and llama.cpp go under C:\AI.

  1. Clone the repo. The later steps run from inside it.

    git clone https://github.com/ramaraihan12/local-ai-coding-setup
    cd local-ai-coding-setup
  2. Unzip llama-b11429-bin-win-cuda-12.4-x64.zip and cudart-llama-bin-win-cuda-12.4-x64.zip from the llama.cpp release tag b11429 into C:\AI\llama.cpp.

  3. Install the hf CLI (uv tool install huggingface_hub) and download the models (checksums).

    $G = "C:\AI\models\gguf"
    hf download unsloth/gemma-4-26B-A4B-it-GGUF --include "gemma-4-26B-A4B-it-UD-IQ3_S.gguf" --local-dir "$G\unsloth\gemma-4-26B-A4B-it-GGUF"
    hf download unsloth/Qwen3.5-9B-GGUF --include "Qwen3.5-9B-IQ4_XS.gguf" --local-dir "$G\unsloth\Qwen3.5-9B-GGUF"
    hf download ggml-org/Qwen2.5-Coder-0.5B-Q8_0-GGUF --include "qwen2.5-coder-0.5b-q8_0.gguf" --local-dir "$G\ggml-org\Qwen2.5-Coder-0.5B-Q8_0-GGUF"
    hf download ggml-org/Qwen2.5-Coder-1.5B-Q8_0-GGUF --include "qwen2.5-coder-1.5b-q8_0.gguf" --local-dir "$G\ggml-org\Qwen2.5-Coder-1.5B-Q8_0-GGUF"
    # optional, only for -Preset qwen36
    hf download unsloth/Qwen3.6-35B-A3B-GGUF --include "Qwen3.6-35B-A3B-UD-IQ2_M.gguf" --local-dir "$G\unsloth\Qwen3.6-35B-A3B-GGUF"
  4. Install Ollama (winget install --id Ollama.Ollama -e), turn off its autostart, set OLLAMA_MODELS=C:\AI\models\ollama, then ollama create qwen3.5-9b:iq4xs -f configs\ollama\Modelfile.qwen35-9b.

  5. Optional: add llama-server.exe and llama-bench.exe in NVIDIA Control Panel > Program Settings once, then python -I scripts\nvidia-sysmem.py --apply (--check only reads).

  6. Pi: npm i -g @earendil-works/pi-coding-agent@1.1.0, winget install BurntSushi.ripgrep.MSVC sharkdp.fd, then pi install git:github.com/obra/superpowers@v7.0.0 and copy the config.

    $A = "$env:USERPROFILE\.pi\agent"; New-Item -ItemType Directory -Force $A | Out-Null
    Copy-Item configs\pi\models.json, configs\pi\settings.json, configs\pi\mcp.json $A
    Copy-Item configs\pi\global-rules.md "$A\AGENTS.md"
    Copy-Item configs\pi\extensions, configs\pi\prompts, configs\pi\mcp-servers $A -Recurse -Force
    uv tool install "duckduckgo-mcp-server[browser]==0.7.0" --with "ddgs==9.16.0"
  7. VS Code: install llama-vscode and copy configs/vscode/settings.json into your user settings.

  8. Add one line to $PROFILE (works in pwsh 7 and 5.1).

    $s = '<clone>\scripts\ai-shortcuts.ps1'; if (Test-Path -LiteralPath $s) { . $s }

Daily use

ai-on            # autocomplete on CPU (:8012) + Gemma (:8080), waits until healthy
ai-fast          # same, with Qwen3.5-9B fully on the GPU instead
ai-pi            # Pi in the current folder; ai-pi -Lite skips skills and MCP for a smaller prompt
ai-status        # servers, loaded models, VRAM, RAM
ai-off           # stop everything; VRAM should read 0 MiB

05 limits

What to keep in mind

  • "Local" means the models. Context7 docs and web search in Pi go over the network (-Lite turns them off).
  • The gate isn't a sandbox. The permission gate is an emergency brake against prompt injection.
  • Windows only. It depends on NVAPI, the Ollama tray app and Windows paths in the gate.
  • The 9B needs review. It's about 4x faster per task, but it left a bug that passed the tests. I use it for small things and review everything it writes.
  • Thinking can eat the budget. It can use the whole 8192-token output budget before any edit. Qwen3.6 failed an A/B task that way.