Experiences with local models for coding

Birgitta Böckeler · rss · 2026-07-08 source ↗ Reader 🖥️

Agent Harness Coding Agents Cognitive Surrender Context Rot & Context Limits Evals Feedback Loops Inference Optimization Local AI

Running small, locally-hosted open models for agentic coding is not yet a plug-and-play substitute for frontier models — it remains frustrating and far below frontier capability, and is viable today only for a narrow class of small, well-scoped, often pre-planned tasks, with the choice of task (more than the model or harness) being the dominant factor that determines whether it works.

practical · mixed — a first-person practitioner experience memo (narrative), scaffolded by an expository evaluation rubric (the 'viability funnel'), carrying an argumentative through-line that reaches a hedged viability verdict — Second of a two-part series on Martin Fowler's site (the first memo covered the enabling factors — hardware, model choice, runtime, harness). Reader/RSS metadata attributes it to 'Martin Fowler' but the content and Reader summary confirm the author is Birgitta Böckeler. She is explicit about the anecdotal, small-N, low-tech-stack-diversity nature of her evidence and repeatedly hedges her generalizations.

Structure

Part	Role
Scope + Viability funnel	Establishes the evaluation rubric — a 7-stage funnel of increasing viability (fits RAM → runs at reasonable speed → handles tool calling → builds functionally correct code → sustains continued conversation/context → handles a larger/complex task → acceptable code quality)
The journey (three phases: manual evals → automated evals → day-to-day use)	Narrates her method and its evolution, framing the whole exercise as a 'roller coaster' with confusing, non-monotonic results
Tasks (Task 1 sort/cumulate bar chart; Task 2 countries bar chart), each with Phase 1 manual + Phase 2 automated results	The evidentiary core — two concretely specified tasks run across models (Qwen3.6 35B, Gemma 4 31B/26B, Qwen Coder Next 80B MoE) and harnesses (OpenCode, Pi)
Day-to-day use + task-characteristic reflections + 'Back to basics' + second perspective (Jigar Jani)	External validation and boundary-setting: extends beyond the structured tasks to real workflows, distils task-selection heuristics, adds a colleague's more-successful counter-case, and reframes the constraint as a benefit
To sum up + default setup + exact model IDs	The hedged verdict plus a fully reproducible configuration

Key terms

agentic coding (vs. auto-complete)
Her explicit scope: the model works through a coding harness, autonomously reading and changing files via tool calling and multi-turn conversation — not single-line completion. This is the harder task on which she judges local models.
viability funnel
Her ordered 7-stage rubric of increasing demand a setup must survive: (1) fits in RAM, (2) runs at reasonable speed, (3) handles tool calling, (4) builds functionally correct code, (5) sustains a continued conversation / more context, (6) handles a larger or more complex task, (7) code quality is acceptable relative to review effort.
small (local) model
Defined operationally by what fits and runs on her own hardware (M3 Max 48GB, M5 Pro 64GB) — quantized (mostly 4-bit) open models, often Mixture-of-Experts with few active parameters (e.g. qwen3.6-35b-a3b = ~3B active of 35B). 'Small' is relative to frontier cloud models, not an absolute size.
manual evals vs. automated evals
Manual = repeating a few chosen tasks by hand across models/configs to feel the user experience; automated = a vibe-coded one-shot eval harness that runs a task N times for more data, with the agent expected to 'one-shot' the problem. The two modes can disagree.
text wall of doom
Her name for the runaway, repetitive output a small model emits when a conversation is continued past a certain point (e.g. Gemma 4 26B when asked for a follow-up refactor); she notes it may be mitigable by enabling a 'presence penalty'.
sensor (for self-correction) Feedback Loops
A feedback channel the agent can call to check and correct its own output (e.g. a browser it can inspect after editing a chart). In the automated eval she gave the agent a browser as its only sensor, and it never called it; she argues expanding sensor use could enable self-correction and raise viability.

Propositions

explicitRunning small, locally-hosted models for agentic coding (not just auto-complete) is not yet a plug-and-play experience; the results are frustrating and often confusing.conf: high
explicitThe choice of task is the single biggest factor determining whether a small local model is viable — driven by task complexity (reasoning), the number of files the agent must read/write (tool calling + context window), and how specific the instructions can be.conf: high
refinesGitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · p12 — SmallCode claims the harness (not the model) is the dominant lever for small-model agentic coding. Böckeler's field experience refines this: task selection is at least as dominant, and even hardware/RAM matters — swapping harness (OpenCode vs Pi) 'did not make much of a difference' on her hardest tasks, and the same model swung wildly in quality between a 48GB and a 64GB machine.
explicitSmall local models handle small, well-defined, often pre-planned changes acceptably, but fall apart on more complex logic.conf: high
refinesGitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · p14 — SmallCode presents locally-run coding agents as a viable adoption path for serious software work. Böckeler refines the scope: viable today only for small, well-scoped, often pre-planned changes; more complex logic 'fell apart', and the overall experience is 'not plug-and-play' and 'very far away' from frontier models.
explicitManual, feel-based evaluation and an automated one-shot evaluation of the same model on the same task can yield contradictory verdicts.conf: high
explicitdevelopedThe same model with identical settings produced markedly different output quality on a 48GB M3 Max versus a 64GB M5 Pro (Task 2: failed 5/7 vs failed only 1/7), an effect she cannot explain and calls a mystery.conf: high
extends ←Viability of local models for coding · p7 — Offers a tentative causal hypothesis (MoE expert-loading differences by available RAM) for the same unexplained 48GB-vs-64GB quality divergence the companion memo reports without diagnosis.
explicitContinued conversation and larger context form a distinct failure boundary: runtimes crash or the model emits runaway repetition ('text wall of doom') as context grows.conf: high
explicitA model can be capable but not realistically runnable: Qwen Coder Next 80B MoE produced a functionally correct implementation in ~2.5 minutes but crashed the runtime on the very next message.conf: high
explicitCumulative or derived computations (e.g. cumulative-percentage x-axis labels) are where small models most frequently struggle.conf: high
explicitSwapping the coding harness (OpenCode vs. Pi) made little difference on the hardest tasks.conf: high
explicitRAM/context is the binding physical constraint: 48GB 'stretches' the 35B model and forces closing other applications, and more code search or bigger files fill the context window and consume precious RAM.conf: high
explicitBash and Python tasks succeeded more often than JavaScript/TypeScript tasks — an impression drawn from a very small, low-diversity dataset.conf: medium
explicitThe agentic coding capabilities of small local models remain very far from what she is now used to with bigger frontier models.conf: high
refinesLocal AI · p1 — Loukides claims local open-weight models are now competitive with frontier cloud models for most production use cases. Böckeler carves out agentic coding as a domain where that does not yet hold — capabilities remain 'very far away' from bigger models — refining rather than denying the general claim.
explicitPrompting small local models re-imposes the discipline of specific, precise instructions that capable frontier models had let developers abandon.conf: high
explicitcorroboratedHer converged default is qwen3.6-35b-a3b at 4-bit (Q_4_KM), reasoning off, max context window, in OpenCode or Pi, reached for only on small, straightforward, often pre-planned tasks.conf: high
supports ←Viability of local models for coding · p13 — The companion memo (published the next day) independently converges on the exact same go-to default: qwen3.6-35b-a3b at 4-bit.
explicitA colleague (Jigar Jani) does almost all of his coding with Qwen 35B MoE (4BIT) on a 48GB MacBook against a real-world Python and React codebase by continuously enhancing his harness with skills (Graphify, Understand Anything), while stressing that code review is essential.conf: high
explicitWorking with weaker local models restores comprehension and careful in-the-moment review — an antidote to the constant rework and surprises that came from giving strong models lots of autonomy and 'surrendering' to them.conf: high
implicit⚡ disputed & corroboratedBecause task and evaluation method so heavily determine outcomes, benchmark numbers reported without task specification and methodology are close to meaningless for judging small local models.conf: high
supportsLocal AI · p9 — Loukides argues published benchmarks are an unreliable basis for model selection and domain-specific evaluation is necessary. Böckeler's evidence supports this strongly: 'which task to give makes a huge difference', and her manual and automated evals reached contradictory verdicts on the same model.
contradicts ←Alexander Haneng's Post · p1 — This post treats an unbenchmarked, task-unspecified 'on par with ChatGPT-4o-mini' claim as meaningful and shareable, directly at odds with this corpus's own finding that such claims are close to meaningless without task and methodology.
supports ←NVIDIA Personal AI Router Distributes AI Tasks Across Local Compute · p12 — The disclaimed, single-configuration 2x demo figure is a direct instance of the benchmark-without-methodology problem this proposition names for local-model evaluation generally.
implicitSmall-model coding evaluations may be memory-pressure- and machine-dependent, and therefore not reproducible across hardware — which would undermine any attempt to standardize a single-machine benchmark for these models.conf: medium
implicitProviding self-correction sensors that the agent actually uses (e.g. a browser it can inspect after editing) could materially raise small-model viability; some one-shot failures are missing feedback loops rather than raw incapability.conf: high
extendsThree more static code analysis sensors · p2 — Thematic extension of Böckeler's own sensor / self-correction program (that extract is about *static* code-analysis sensors whose custom error messages steer agents; here she invokes a *runtime* browser sensor the agent never called). Same underlying idea — self-correction feedback changes agent behaviour — extended to small local models as a lever for viability.
implicitAs model capability drops, more of the burden shifts to the harness and the human — deliberate harness engineering (skills, code-search/understanding tools) is what makes near-all-local coding workable.conf: medium
implicitViability is a relation among model, task, harness, and hardware rather than a property of the model alone, so a personal, intuition-building practice ('knowing when to reach for it') is currently the only reliable selection method.conf: medium

Arguments

Small local models for agentic coding are viable today only for a narrow class of small, well-scoped, often pre-planned tasks, and are not a general substitute for frontier models.

Explicit premises:

The choice of task is the biggest single factor determining viability.
On more complex tasks the models fall apart, crash on continued context, or must be given up on ('gave up after 11 minutes').
Even where they succeed, capability is 'very far away' from what bigger models deliver.
The setup is 'not plug-and-play' and requires careful, hands-on work.

Implicit premises:

Viability is task-relative, not an absolute model property — so a verdict is only meaningful once the task class is fixed.
The reader's job is to calibrate expectations to the task, because the same model swings between success and failure with task, machine, and method.
Because published/aggregate benchmarks abstract away the task, they cannot answer the viability question the way concrete, specified tasks can.

Inferences:

The 48GB-vs-64GB same-settings quality divergence implies small-model evals may be non-reproducible across hardware, undermining standardized single-machine benchmarking — she flags it as a mystery but does not draw this methodological conclusion.
If missing/unused sensors explain many one-shot failures, then reported one-shot 'failure rates' understate achievable viability and conflate model limits with harness gaps — she implies but does not state this.
The 'detox' benefit could be turned into a deliberate recommendation (use weaker models as a comprehension/friction tool), but she leaves it as a personal reflection rather than advice.

Narratives

Task 1 — asking the models to rename a chart, sort its bars descending, and add cumulative-percentage x-axis labels every 10th bar; Qwen Coder Next 80B MoE one-shot it in 2.5 min then crashed on the next message, Qwen3.6 35B and Gemma 4 31B needed 15 min of back-and-forth on the cumulative percentages, and Gemma 4 26B succeeded fully but then produced the 'text wall of doom'.
→ Concretely demonstrates the funnel's middle stages (functional correctness and continued-conversation limits) and shows that 'success' and 'usable' are different things.
Task 2 — adding a horizontal 'Countries' bar chart (top 10 + 'Other') to a single access-log visualisation script; the harnessed agents burned 8-12 minutes and were abandoned ('Gave up after 11 minutes'), yet pasting the whole 450-line file into a plain LM Studio chat with no harness produced a working solution in ~6 minutes.
→ Shows that harness overhead and code-search can hurt more than help for a single-file task, and that a 'go old school' whole-file paste can beat an agentic setup.
Re-running the Task 2 automated eval on the 64GB M5 Pro after the 48GB M3 Max — same model, same settings — dropped failures from 5/7 to 1/7, a 'vast difference in quality' she calls a mystery.
→ The standout anomaly: implies that hardware/memory-pressure, not just model choice, governs output quality, quietly destabilizing any single-machine benchmark.
The three-phase 'roller coaster' journey — manual evals to feel the UX, then a vibe-coded automated eval setup after Thoughtworks colleagues shared input, then day-to-day use of Qwen3.6 35B MoE on real work and personal tasks (bash/Python scripts, website content, small changes, a game whose complex logic fell apart).
→ Frames evaluation itself as iterative and setup-sensitive, and maps which everyday task types actually work.
The 'back to basics / detox' reflection — enjoying the return to weaker capabilities because it lowers the temptation to check out, increases signal about what helps, and restores real review instead of surrendering to strong-model autonomy and paying for it in later rework.
→ Carries the secondary thesis that the constraint of weak models is itself a benefit to comprehension and craft.
Colleague Jigar Jani does almost all his coding with Qwen 35B MoE (4BIT) on a 48GB MacBook against a real-world Python/React codebase, improving as he enhances his harness with skills like Graphify and Understand Anything, while insisting code review is essential.
→ A more-successful counter-case that bounds her own pessimism and shifts credit toward harness engineering and human review.

Questions addressed

✅ Which kinds of coding tasks can small, locally-run models actually handle for agentic coding right now?
Small, well-scoped, single-/few-file, often pre-planned changes — plus many bash/Python scripts — work acceptably; tasks needing much code search, large context, cumulative/derived logic, or continued conversation tend to fail or become not-realistically-runnable.
✅ Is local small-model agentic coding viable as a day-to-day practice?
Not plug-and-play and very far from frontier models, but usable enough that she integrates qwen3.6-35b-a3b for narrow, pre-planned tasks and keeps building intuition about when to reach for it.
⭕ How should one evaluate small local models for coding?
Partially: her viability funnel plus carefully specified tasks give structure, but manual and automated evals disagreed and she does not resolve which is the truer signal or how to reconcile them.
⭕ Why did the same model with the same settings produce vastly different output quality on the 48GB vs. the 64GB machine?
Unresolved — explicitly left as a 'mystery'; she does not investigate memory-pressure, runtime, or quantization causes.

Gaps

incompleteThe 48GB-vs-64GB same-model, same-settings quality divergence is reported but never investigated. For a rigorous adopter this is the crux: if output quality depends on RAM headroom/memory pressure rather than model settings, then any single-machine small-model benchmark is untrustworthy and non-transferable.
Read next: Controlled experiments holding model + quantization + settings fixed while varying available RAM headroom; llama.cpp / MLX / LM Studio documentation on memory-pressure, offload, and non-deterministic behavior under constrained RAM; work on eval reproducibility across hardware.
incompleteThe manual-vs-automated eval divergence is described but not diagnosed. It is unclear whether manual iterative 'success' merely reflects the human closing the loop through follow-up messages while the automated one-shot exposes the model's real ceiling — i.e. whether the two modes measure different things.
Read next: Literature comparing one-shot vs. multi-turn agent benchmarks; studies on human-in-the-loop correction inflating apparent capability; self-correction/sensor studies (her own 'sensors for coding agents' work).
incompleteNo systematic cost/latency/throughput accounting. There are vivid anecdotes ('gave up after 11 minutes', '~6 minutes', '2.5 minutes') but no tokens/sec, no minutes-per-task distribution, no download-time, electricity, or RAM-opportunity-cost figures — so the practical economics of the setup remain qualitative.
Read next: Local-inference throughput benchmarks (tok/s by model/quantization/hardware); total-cost-of-ownership analyses for local vs. cloud coding models; latency-vs-quality tradeoff studies for quantized MoE models.
incompleteThe apparent tech-stack effect (Bash/Python succeeding more than JavaScript/TypeScript) is asserted as an impression on a tiny, low-diversity dataset (two main JS/TS tasks) with confounds she names but does not control.
Read next: Larger multi-language evals of the same small models; analysis of training-data composition by language; whether the effect is language difficulty, task difficulty, or harness/tooling maturity.
incompleteThe harness's contribution is under-explored and internally in tension: she found swapping OpenCode vs. Pi made little difference, yet Jigar Jani's near-all-local coding depends on continuously enhancing his harness with skills. How much of viability is raw model vs. harness engineering (skills, code-search/understanding tools, self-correction sensors) is left open.
Read next: SmallCode-style harness-engineering studies for small models; her own harness-engineering framework; controlled comparisons isolating harness features (progressive disclosure, tool routing, skills) at fixed model.
incompleteThe reasoning-on/off tradeoff is unexamined. Her default runs with 'reasoning off', yet the failures she reports include 'very long reasoning chains followed by excruciatingly slow attempts to finally edit the file' — the effect of thinking mode on tool-calling reliability, latency, and RAM for small models is never systematically tested.
Read next: Model documentation on thinking/reasoning modes for tool-calling agents; studies on reasoning-token cost vs. task success for small quantized models; effect of presence penalty and sampling settings on runaway output.
closed byViability of local models for coding — This memo directly tests reasoning on vs. off in an automated eval and finds reasoning off is faster and performs the same to slightly better — a preliminary empirical answer to the reasoning on/off tradeoff the companion memo leaves unexamined.

Closes gaps elsewhere

Local AI · g3 — Partially. Loukides' g3 notes the inference-throughput disadvantage of local models is underexplored. Böckeler adds first-hand latency/throughput observations (a speed 'smoke test' as funnel stage 2, 'excruciatingly slow' edits, minutes-per-task), but gives no tokens/sec figures, so the disadvantage is illustrated qualitatively rather than quantified.
GitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · g3 — Partially. SmallCode's g3 asks the general question of at what task complexity a small model falls off and where the harness breaks. Böckeler supplies concrete, hands-on failure boundaries — cumulative/derived computations, continued-conversation runtime crashes, the 'text wall of doom', and complex game logic falling apart — from OpenCode/Pi/LM Studio setups (not SmallCode's own harness), so it answers the general question, not SmallCode's specifics.
GitHub - Doorman11991/smallcode: AI coding agent optimized for small LLMs. 87% benchmark with 4B-active model. · g4 — Partially. SmallCode's g4 flags missing total-cost-of-ownership discussion (hardware, latency, RAM). Böckeler reports concrete hardware (M3 Max 48GB, M5 Pro 64GB), RAM pressure (48GB 'stretches' the 35B model, must close other apps), and latency anecdotes (2.5 min success, ~6 min, 'gave up after 11 minutes'). Still qualitative and specific to her setups rather than SmallCode's, so it narrows but does not close the accounting.