Every AI system ever built began the same way.
Someone looked at a domain, medicine, law, language, chess, driving, coding, and made a list. Of objects. Of properties. Of relationships. Of rules. Of patterns. They took the living, backgrounded, breakdown-navigating reality of human expertise and drew a boundary around the parts they could see and name.
Then they built inside that boundary.
This is not a criticism of the people who built these systems. It’s the only place you can build from. You can only formalize what you can see. And you can only see what your current interpretive framework makes visible. Which means every formal system begins with an act of selection that is simultaneously an act of exclusion, and the exclusion is invisible because it happened in the background before the formalization started.
Winograd and Flores called this the formalization trap. Not a mistake you can avoid with more care or better methodology. A structural condition of building formal systems to represent domains that are ultimately grounded in unformalized background.
You cannot get outside the trap by being smarter about going in. The trap is the entry point.
Here’s how the trap works in practice.
A system is built to handle the central cases of a domain, the situations the designers could clearly see and articulate. It performs well there. Impressively well. Users adopt it. Depend on it. The central cases become routine and the system handles them better than humans who are now out of practice because the system has been doing it.
Then an edge case arrives. A situation that falls outside the original formalization. Not dramatically outside, close enough to the center that the system engages with it confidently. But the features that make it genuinely different from the central cases live in the background. In the unformalized. In exactly the place the system cannot reach.
The system doesn’t know it’s at the edge. It has no mechanism for knowing. It processes the edge case using the categories built for the central cases and produces an output that is confident, fluent, and wrong in ways that are difficult to detect because the wrongness lives in the background that the user also can’t easily articulate.
This is worse than obvious failure. Obvious failure triggers breakdown. Breakdown triggers inquiry. Inquiry surfaces the background. The system that fails visibly teaches you something.
The system that fails invisibly, that produces confident wrongness at the edges, does something more dangerous. It colonizes the background without announcing itself. It replaces the user’s developing judgment with outputs that feel like judgment but aren’t. And because the user never encounters the productive friction that would have built their competence, they have no independent standard against which to detect the invisible failure.
The formalization trap has a second jaw. The first jaw is that you can’t formalize the background. The second jaw is that dependence on the formalization erodes the background in the user. Both jaws close simultaneously as adoption deepens.
Winograd and Flores saw the first jaw clearly in 1986. They were writing before the second jaw became fully visible. We’re living inside both jaws now.
Large language models changed the geometry of the trap without escaping it.
Previous AI systems were built on explicit formalizations, hand-crafted rules, ontologies, knowledge representations. Their failures were often traceable back to specific gaps in the formalization. You could, in principle, find where the boundary was and understand why the edge case fell outside it.
LLMs absorbed formalization implicitly, statistical patterns extracted from the full texture of human language. The boundary is no longer a wall you can find and examine. It’s a probability distribution that fades at the edges without announcing that it’s fading. The system doesn’t switch from confident to uncertain as it moves from center to edge. It stays confident. The confidence is baked into the architecture.
Which means the second jaw closes faster and quieter than it ever did before.
Users who depend on LLM outputs for domains they’re learning don’t develop the background that would let them detect edge failures. The formalization trap now operates without leaving visible seams. Without the obvious breakdown that would trigger inquiry. Without the productive friction that builds competence.
And here’s what makes this specific to the Flores framework rather than just a general AI safety concern, the problem isn’t that the outputs are sometimes wrong. Everything is sometimes wrong. The problem is that the mechanism by which humans develop the background to detect wrongness is being systematically undermined by the very fluency that makes the outputs useful.
The trap doesn’t just catch the AI. It catches the human who depends on it.
Flores in 1986 was trying to prevent this. Not by opposing technology, he spent his later career building technology. By insisting that design start from an honest account of what humans are and what background does.
We didn’t listen then. We’re paying the tuition now.
@ Lee Greene