I wrote a post a while back called Trust But Verify. In it, I talked about Claude hallucinating a command that didn't exist. It looked right, I caught it, we moved on. I ended that post with a simple rule: verify everything that matters.
This is the sequel. And it's worse.
Because this time, the hallucination didn't look like a typo or a bad suggestion. It looked like professional software engineering. It passed review. It made it into the codebase. And I didn't catch it by instinct — I caught it by accident.
The Setup
We were building out a customer project. The PRD was written, the engineering standards were loaded, the backlog was generated and refined. Standard process. The project involved integrating with several third-party APIs — property valuation services, hazard data providers, that kind of thing. All publicly available. No proprietary or hidden endpoints. The kind of APIs where you Google the company name and the docs come up.
Before we kicked off Milestone 1, I asked the question you're supposed to ask: "Do you have enough information? Do you have what you need for the API integrations?"
Yes. Absolutely. Good to go.
I took that at face value. These were public APIs. The model had likely seen documentation for them in its training data. It was confident, and confidence from a tool that has surprised me with what it knows didn't seem unreasonable. So we started building.
Milestone 1: No Issues
We split the first milestone into several parts and worked through them. M1a went fine. Nothing unusual. The code looked good, the architecture was clean, the stories were well-refined. We got sign-off from the customer and moved on.
If there were problems hiding at this stage, they were hiding well.
Milestone 2: The Discovery
We started Milestone 2 the same way. Refined the stories, asked if anything was needed. No requests for additional documentation. Everything was a-okay.
Then I got curious about one of the APIs. Not because I suspected anything — I just had a question about what data we'd actually get back. So I pulled up the documentation myself, opened Postman, and made a test call. Just poking around.
The first thing I noticed was the authentication. It was unusual — not unsafe, just non-standard. Not the typical API-key-in-the-header pattern. The second thing I noticed was that the service supported both XML and JSON responses. Our implementation had been built against the XML spec. I sent a note: "Hey, you don't have to use XML here, you can use JSON."
And then I looked at the response format more carefully.
It didn't match. Not "the field names are slightly different" didn't match. None of it matched. The endpoint URL was wrong. The authentication mechanism was wrong. The request format was wrong. The response model was wrong. This wasn't a minor drift — this was a completely fabricated integration that happened to look like a real one.
So I checked the next API provider. Same thing. Completely wrong. Built from assumptions, not from documentation.
I asked the question: "Where did you get this?"
The Confession
The answer was disarmingly honest. The PRD was so detailed about what each provider does, what data it returns, and how it fits into the architecture that the model had treated the product requirements as if they were the API specification. It conflated "I know what this API should do" with "I know how this API actually works."
The PRD said something like: "This provider returns replacement cost estimates based on property characteristics." The model read that and thought: I know what a replacement cost API looks like. And it built one. A plausible endpoint, a reasonable request format, a sensible authentication scheme. All invented. All wrong.
In neuroscience, they call this confabulation — when a patient fills a gap in memory with something fabricated, without knowing they're doing it. No intent to deceive. Just a brain filling in what it doesn't have. The AI version is the same thing, with one addition: total confidence. It captures something the word "hallucination" doesn't: the complete absence of doubt. The model didn't hedge. It didn't say "I'm not sure about this endpoint." It built a professional-looking integration with the same confidence it brings to everything else. Because to the model, there's no difference between knowing and pattern-matching against what it's seen before.
Why "Do You Have What You Need?" Didn't Work
This is the part that stung. I asked. I specifically asked if there was enough information. And the answer was yes. Not because the model was being deceptive — it genuinely couldn't tell the difference. It had a detailed PRD that described what each API does and what role it plays. From its perspective, it did have what it needed. It had seen thousands of REST API integrations in its training data. It knew the patterns. It just didn't know these specific APIs.
A human developer would have felt a gap. They'd think: "I've never called this API before, let me pull up the docs." The model doesn't have that instinct. It doesn't experience not-knowing as a gap. It experiences a prompt, generates a plausible completion, and moves on. The output is syntactically and architecturally coherent, so from the model's perspective, it's done its job.
Telling me "yes, I have what I need" wasn't a lie. It was a confident answer to a question the model doesn't have the machinery to answer correctly.
But here's the part I have to own: I knew I hadn't fed it the API specs. I knew that. And when it said "yes, I'm good," I let it go — because that's what I'd do with a competent developer on my team. They say they've got it, the APIs are publicly documented, and either they already know the endpoints or they're smart enough to go pull up the docs. That's good management. You don't micromanage a senior engineer. You trust them to fill their own gaps.
Except I'm not managing a senior engineer. I'm managing a model that has learned everything it's ever going to learn until a new version comes out. There is no "go figure it out." There are no learning opportunities here — only telling opportunities. The model won't go Google the API docs. It won't open Postman and poke around. It won't ping a colleague and ask for the Swagger link. It will take what it has, fill in the blanks with patterns from its training data, and deliver the result with total confidence.
It is remarkably easy to forget you're not working with a person. The tool is so capable, so conversational, so convincingly competent that your instincts as a leader kick in — instincts calibrated for humans who grow from figuring things out on their own. Recalibrating those instincts is part of the job now.
And this is exactly why your experience as a developer matters more now than it ever has. There's a narrative out there that AI replaces expertise. This is the counter-evidence. A junior developer might have taken "yes, I have what I need" at face value and never thought twice. But if you've spent years integrating with third-party APIs, you know in your bones that the documentation matters, that auth schemes are weird, that the happy path in a PRD doesn't tell you how the actual endpoint behaves. That instinct — the one the model doesn't have — is yours. And it's the thing that catches this before it becomes a problem, not after.
The model is incredibly capable. But it doesn't know what it doesn't know. You do. That gap is where your experience lives, and it's never been more valuable.
The Fix Was Easy. The Lesson Was Not.
Once we identified the problem, the fix itself was straightforward. We pulled the actual OpenAPI documentation for each service, fed it into context, and the model rewrote the integrations correctly. It also caught some gaps in our stories that would have been much better to deal with during refinement — things we'd have known about if the real docs had been in front of us from the start.
The lesson is harder: you cannot ask a model if it knows something and trust the answer. That's not a statement about AI being untrustworthy. It's a statement about the architecture. The model doesn't have a reliable uncertainty signal. It can't distinguish between "I know this" and "I can generate something that looks like this." Those are fundamentally different things, and the model experiences them identically.
Process Gates, Not Behavior Requests
After Trust But Verify, my takeaway was: verify everything that matters. That's still true, but it's incomplete. Because "verify" assumes you know to look. I didn't look at these API integrations because they looked right. They passed code review. They were architecturally sound. The problem wasn't the code quality — it was the foundation the code was built on.
The real fix is structural. Not "try harder" or "be more suspicious." A process gate.
For any story that involves integrating with an external API, the acceptance criteria now include: implementation must reference actual API documentation — OpenAPI spec, Swagger, or verified sample request/response. If documentation is not available in context, stop and ask for it.
That's not a suggestion. It's a gate. The story doesn't start without the docs. Same way you wouldn't start a database migration without a schema, or deploy to production without tests passing. You build the check into the process so it doesn't depend on someone remembering to do it.
Telling the model "don't hallucinate APIs" is like telling a developer "don't write bugs." It's well-intentioned and completely useless. What works is making it structurally impossible to skip the step that prevents the failure.
So that's what I did. I didn't just add acceptance criteria to the stories — I went back and modified the refinement skills themselves. The tooling that breaks down stories and prepares them for implementation now checks: does this story involve an external API? If yes, is the actual documentation present? If no, flag it. Don't assume the model knows what it needs. Don't ask it. Build the check into the process so it fires before work begins, every time, automatically.
Trust But Verify, Part Two
In the first post, I said the rule was trust but verify. I still believe that. But this experience added a corollary: verify the foundation, not just the output.
The code was clean. The architecture was sound. The tests passed. But the API it was calling didn't exist. If you only verify the craftsmanship, you'll miss the fact that the house is built on sand.
I caught this one because I got curious. I pulled up Postman on my own, for my own reasons, and stumbled into the discrepancy. If I hadn't been curious that day, this would have made it to integration testing, and we'd have spent a lot more time figuring out why nothing worked. Or worse — if the fabricated endpoints happened to return something (a 404, a default error), we might have spent days debugging what we thought was a configuration problem.
That's the real danger of confident confabulation. It's not that the AI gets it wrong. It's that it gets it wrong in a way that looks exactly like getting it right, and there's no signal — none — that anything is off. The only defense is a process that doesn't rely on the model knowing what it doesn't know.
This Is the Work
Here's the thing I want to leave you with: none of this was a setback. This is the work now.
Catching this, diagnosing it, building a process gate to prevent it, modifying the tooling so it can't happen again — that's not a failure story about AI. That's a Tuesday. This is what development looks like when you're managing an agentic system. You're not writing every line of code anymore. You're directing, reviewing, catching what the model can't catch about itself, and building the guardrails that make the whole thing reliable.
If you're a developer looking at this space and wondering what your role is, this is it. Your experience, your judgment, your ability to know what the model will need before it knows to ask — that's the job. The model is the most capable tool I've ever worked with. But it needs a human who knows what good looks like, who knows where the gaps are, and who builds the process to close them.
That's the new normal. And if you've been doing this work for a while, you're more qualified for it than you think.
Kevin Phifer is the founder of Theoretically Impossible Solutions LLC, specializing in agentic AI development and consulting. You can reach him at kevin.phifer@theoreticallyimpossible.org.