Viability of local models for coding

Martin Fowler (martin@martinfowler.com) — memo by Birgitta Böckeler · rss · 2026-07-07 source ↗ Reader 🖥️

Agent Harness Agent Tool Management Coding Agents Context Rot & Context Limits Inference Optimization Local AI

The practical viability of local models for agentic coding is not a property of the model alone but an emergent, hard-to-predict outcome of interacting hardware (RAM, processing power, memory bandwidth), model (parameters, architecture, quantization, reasoning) and harness (context overhead, tool-calling schema) factors — which is why local coding assistants are still not a reliable 'plug and play' option for developers, even though specific configurations already deliver acceptable results for narrow tasks.

practical · expository — The memo is structured as a systematic breakdown of the factors that determine whether local models are usable for agentic coding — mostly expository (defining and explaining each factor) with an argumentative undertone driving toward a practical recommendation (a go-to local setup) and a practical verdict ('not yet plug and play').

Structure

Part	Role
Scope and hardware disclosure (intro)	Defines 'viability' as usability for agentic coding (not just autocomplete) and readiness for developers who don't want to tune specs and tooling; discloses the two test machines.
Summary diagram and bullet overview (Runnability, Response speed, Tool calling, Quality of outcomes)	Framework/context — previews the four outcome dimensions and the causal graph of factors feeding them, before the deep dives.
Per-factor deep dives (RAM, Processing power, Memory bandwidth, Number of parameters, Reasoning, Tool calling, Format, Quantization, Architecture, Context window, Runtime, Harness)	Evidence — for each factor, states what it impacts and reports the author's own hands-on experience, building the case that viability is multi-factorial and interdependent.
List of models used	Method transparency — grounds the experiential claims in a concrete, named set of models and quantization levels.
Coming up / conclusion	Conclusion — states the overall verdict ('not ready for plug and play') and names a converged personal default (Qwen3.6 35B MoE), previewing a follow-up memo with task-level detail.

Key terms

Viability
Whether a local model is usable for genuinely agentic coding (not just autocomplete) by developers who are not willing to dive into specs, extra tooling and tweaks — a bar that includes ease of setup, not just raw capability.
Runnability
Whether a model's weights (plus context) fit into available RAM/VRAM without crashing or dropping to unusably slow speeds — a binary-ish threshold distinct from speed or quality.
Harness Agent Harness
The coding agent tool (OpenCode, Pi, Claude Code) plus its own configuration (skills, MCP servers) wrapped around the model — a source of context-window overhead and tool-call schema variance that becomes a bigger constraint locally than in the cloud.
MoE (Mixture of Experts) architecture
A model architecture with a large total parameter count that activates only a subset of weights per inference step, so a MoE model of a given size needs less RAM and runs faster than a dense model of the same size — and, per the author's tentative hypothesis, may load more experts on machines with more RAM, improving quality.
QAT (Quantization-Aware Training) Inference Optimization
Quantization simulated during training (as opposed to applied after training) so as to better preserve model quality at a given compression level; mentioned as 'the newest buzz' but not yet personally tested by the author.

Propositions

explicitdevelopedRAM (effectively unified/VRAM capacity) is the core physical constraint on whether a local model can run at all; exceeding it crashes the runtime or drops speed to unusable levels.conf: high
extends ←Generative AI in the Real World: Local Voice AI with Pete Warden · p4 — Supplies concrete quantization arithmetic (9B model at 8-bit ~9GB; Q4 schemes under 4GB) for the general claim that RAM/VRAM capacity is the core physical constraint on local-model viability.
refines ←NVIDIA Personal AI Router Distributes AI Tasks Across Local Compute · p7 — PAIR explicitly leaves the per-node RAM/VRAM capacity ceiling this proposition names untouched — it relieves a different constraint (concurrent-request contention across nodes) rather than the single-machine capacity limit itself, sharpening the boundary of what 'distributing local AI work' can mean.
explicitResponse speed for a given model is jointly determined by memory bandwidth, processing power, quantization level and format, with no single factor dominating.conf: high
explicitReliable tool calling, more than raw output quality, is the key differentiator between agentic coding viability and mere autocomplete viability for small local models.conf: high
explicitSmall local models can often self-recover from malformed tool calls, which makes imperfect tool-calling compliance survivable for agentic use.conf: high
explicitTurning reasoning off can make a model both faster and equal-to-slightly-better in output quality than with reasoning on, contradicting the assumption that more reasoning always helps complex tasks.conf: medium
explicitMoE architecture offers a better balance of RAM footprint, speed and quality than a dense model of similar parameter count.conf: medium
explicitThe identical model with identical settings can produce a meaningfully better coding outcome on a machine with more available RAM, even when the underlying cause is unclear.conf: high
extendsExperiences with local models for coding · p5 — Offers a tentative causal hypothesis (MoE expert-loading differences by available RAM) for the same unexplained 48GB-vs-64GB quality divergence the companion memo reports without diagnosis.
explicitThe context window size that local runtimes default to is too small for agentic coding; at least 32K, and often 64K, is needed.conf: high
explicitHarness overhead (system prompt size, number of tools, active skills/MCP servers) consumes context budget that is scarcer locally than in cloud setups, making harness choice a resource decision as much as a UX one.conf: high
explicitDifferent coding harnesses use inconsistent parameter names for the same basic tool operations (e.g. edit-file arguments), and this schema variance itself contributes to small models' tool-calling failures.conf: high
supportsGitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · p3 — Identifies inconsistent tool-call parameter naming across harnesses as an additional, concrete source of the malformed/heterogeneous tool calls SmallCode's harness is engineered to absorb.
explicitThe perceived user experience of the model runtime (e.g. LM Studio) matters to real-world local-model viability as much as raw technical performance.conf: medium
explicitcontestedLocal models are not yet ready for a 'plug and play' experience for developers unwilling to invest significant tuning effort.conf: high
contradictsLocal AI · p1 — Fowler's hands-on agentic-coding trial ('hit and miss', 'nowhere near' frontier capability, not plug-and-play) directly contradicts Loukides' broader claim that local open-weight models are already competitive with frontier cloud models for most production use cases.
explicitDespite that overall verdict, a specific configuration (Qwen3.6 35B MoE, quantized, reasoning available, generous context) is good enough that the author has adopted it as her personal local go-to model.conf: high
supportsExperiences with local models for coding · p14 — The companion memo (published the next day) independently converges on the exact same go-to default: qwen3.6-35b-a3b at 4-bit.

Arguments

Local models are not yet a plug-and-play option for developers who don't want to spend significant time tuning them. strength: medium

Explicit premises:

Quality of outcomes was 'very hit and miss'.
Tool calling was tricky and models often failed at it.
Many interacting factors (hardware, model, harness) each require separate tuning decisions.
Even carefully chosen setups crashed unpredictably (e.g. a 48GB model on a 64GB machine 'went great at first, but then it quickly crashed').

Implicit premises:

'Plug and play' is being judged against the near-zero-configuration UX bar set by cloud/frontier model products.
Typical developers have low tolerance for spec-reading, tooling setup and iterative tuning.

Inferences:

If the blocker is largely tuning burden rather than raw capability, better defaults and packaged configurations (not only bigger/better models) could close much of the remaining gap.

Reasoning is not always necessary for small local models on coding tasks, and can sometimes be counterproductive. strength: low