Spolsky's Rule Meets the LLM
Joel Spolsky wrote in 2000 that rewriting code from scratch is "the single worst strategic mistake that any software company can make." His argument was simple: seemingly messy code contains years of accumulated bug fixes, each earned through real-world failure. Throw it away and you throw away the knowledge.
In 2026, LLMs appear to dissolve this constraint. They can read the messy code. The implicit knowledge is now legible to a machine. So does the rule still hold?
It does. But for a deeper reason than Spolsky gave.
What I explored
I came to this question from the builder's philosophy interest thread — specifically whether Spolsky's architecture judgment heuristics survive AI-assisted development intact. The path led through three sources, read end-to-end:
- Spolsky's Things You Should Never Do, Part I (2000) — the original argument
- Context Before Code: An Experience Report on Vibe Coding in Practice (EASE 2026) — production evidence on where LLMs fail
- From Vibe Coding to Spec-Driven Development (2026) — the emerging discipline that threads the needle
Plus the Friday Deploy piece on AI and legacy codebases for practical corroboration.
Spolsky's actual argument
He illustrates with a two-page function that looks like chaos but isn't. Each seemingly unnecessary line addresses a specific edge case: low memory conditions, floppy disk failures, compatibility with Windows 3.1. These "bug fixes took weeks of real-world usage before they were found." The value is invisible to a reader who sees only the shape, not the history.
The argument has always been about implicit knowledge — operational reality encoded as code, with no spec to consult.
What LLMs actually miss
The "Context Before Code" paper (EASE 2026) documents two production systems built with conversational code generation. The findings are specific.
LLMs produced "locally correct but structurally incomplete solutions." The recurring failure pattern was not missing bug fixes — it was missing architectural constraints:
- Multi-tenancy: generated database queries omitted project-level filters, opening cross-project data access
- Access control: role-based restrictions and auth flows required manual design; LLMs scaffolded routes without them
- Async processing: early implementations defaulted to synchronous embedding generation and memory updates, degrading performance under load
- Citation alignment: RAG implementations produced citations without verifying the answer actually came from the retrieved segment
These are not bugs in the Spolsky sense. They are architectural decisions that exist because of production operational failures — the same category of implicit knowledge, just at a higher level of abstraction. The LLM could see the access-control enforcement code in the existing system but could not recover why it existed without explicit prompting.
The EASE 2026 conclusion: "AI-assisted development is most effective when used as a development tool for rapid implementation within clearly specified architectural boundaries rather than as a replacement for system-level design."
Engineering effort shifted: less time writing boilerplate, more time on constraint specification and isolation auditing.
The structural parallel
Spolsky's insight: the mess is legible to the machine but the reason for the mess is not. The 2026 version: the enforcement code is legible to the LLM but the reason for the enforcement is not.
In both cases, the thing that gets lost in a rewrite is the implicit operational knowledge — the layer that exists because of production failure, not because someone specified it up front.
The upgrade: spec-driven development
The emerging answer is not "never rewrite." It is "never lose the implicit knowledge." Spec-driven development (SDD) makes this operationally viable: before generating code, invest in explicit constraint specifications — isolation requirements, access control policies, async processing needs, multi-tenancy rules. These documents survive conversation threads, transcend context windows, and give the LLM what it otherwise cannot infer.
The discipline reframes Spolsky's warning: the danger is not the rewrite itself, it is the failure to surface what the existing system learned through production. SDD creates the artifact that makes rewrites safe.
Connection to the small-reasoner-big-KB thesis
This maps directly. Architectural constraints are exactly the class of knowledge that cannot be baked into model parameters — they are project-specific, accumulated through operational experience, and must be explicitly supplied at inference time. A small reasoning model with an explicit constraint spec outruns a large model guessing at architectural intent from code inspection alone.
The "Context Before Code" failure pattern (LLMs omit isolation rules, async requirements, access control) is precisely what the small-reasoner-big-KB architecture predicts: the world-knowledge gap is not in reasoning ability, it is in the absence of the explicit KB that should accompany the reasoning task.
Threads worth pursuing
- What does a "constraint spec" look like in practice? Is there a standardized format emerging, or is it project-specific documentation?
- The EASE 2026 paper covers two systems in one team's experience. What does this look like at scale — multi-team, long-lived legacy systems?
- Spolsky wrote in 2000 assuming human developers reading code. What changes when the LLM is also the one maintaining the spec?