The demo is the easy part: exceptions, evals and evidence

Every AI system demos beautifully. That's not a compliment to the systems - it's a property of demos. A demo follows the happy path: the clean input, the expected format, the case everything was built around. Production sends everything else.

There is one way a workflow goes right, and there are a thousand ways it goes wrong. Almost all of the value of an automation sits in how it handles the thousand - and almost none of that is visible in a demo. So the interesting question when you're shown one is never "does it work?" It visibly does. The question is: what happens when the input isn't like that?

The exceptions are the product

Think back to what a real workflow looks like once you've mapped it honestly: duplicates, missing fields, formats that follow no rules, the case that fits no pattern. That's not noise around the workflow. In most operations, handling those cases is the workflow - the clean cases were never the hard part.

An automation built for the happy path meets that input distribution and does one of two things: it fails loudly, which is annoying, or it fails quietly, which is worse. The quiet version keeps producing output that looks plausible while being wrong, and degrades for weeks before anyone notices.

Building for the unhappy paths is concrete engineering, not a mindset:

• Structured output, validated. The model returns data in a defined schema, and everything it returns gets checked against that schema before it touches anything downstream. Free-form text into a business process is how plausible nonsense spreads.
• Failure modes enumerated up front. The exception list from discovery becomes a literal checklist: what does the system do when the attachment is missing, the amount doesn't parse, the sender is unknown, the confidence is low? "It won't happen" is not one of the options.
• Recovery, not just detection. Retry where retrying is safe, route to a person where it isn't, and never process the same thing twice by accident.

Evals: turning "it seems to work" into evidence

A deterministic system can be signed off by inspection - read the code, run the tests. A system with a model in it can't, because the same input phrased differently can produce different output. So it needs a different kind of proof: evaluation.

The mechanics are less exotic than the word. You collect a set of real cases with known correct answers - past emails and how they were actually categorized, past documents and what was actually extracted from them. You run the system against them and you count. Then, and this is the part most teams skip, you attribute the failures. A useful eval report doesn't say "82% pass." It says: fifty runs, forty-one passed, and of the nine failures, five were missing input data and four pulled the wrong record. Now you know what to fix. A pass rate without attribution is a mood, not a measurement.

For tasks with a clear right answer, that's the whole game, and history makes it cheap - a few thousand already-handled cases are a ready-made answer key.

Tasks without a single right answer - drafting a quote, writing a summary, composing an email - are harder, and anyone who claims otherwise is selling something. The workable approach: mine your historical output. Five hundred quotes that were actually sent encode what your company considers good - structure, tone, what's always included, what never is. That becomes a golden dataset the system is scored against. But be clear about the ceiling: on subjective work, evals get you a floor, not a guarantee. The gap between "scores well" and "actually right" is covered by exactly one thing, and it isn't more evals.

Oversight is architecture, not training wheels

That thing is people - placed deliberately, not sprinkled on top.

Three pieces belong in the design from day one and never come out:

• An approval gate in front of anything irreversible. Money, customer-facing communication, contracts. The system prepares, a person releases. Designed properly, this is a ten-second review of the uncertain cases, not a bottleneck on everything.
• A feedback loop that feeds corrections back in. Every human override is information: it recalibrates thresholds, refines prompts, grows the eval set. A system without this loop is frozen at its launch-day quality while your business moves.
• An audit trail of everything. Every input, every decision, every action, inspectable after the fact. Partly for debugging, mostly for trust - "what did it do and why" should always have an answer that takes a minute to produce, not a forensic investigation.

Notice what these have in common: none of them are temporary. The popular mental model is that human oversight is scaffolding you remove once the system "proves itself." In every system I've seen hold up over years, it's the opposite - the oversight is load-bearing, and it's why the system holds up. The target state was never zero humans. It's humans spending ten minutes where they used to spend four hours, on the cases where their judgment actually matters.

What to ask instead of watching the demo

If someone - me included - wants to put an AI system into your operation, the demo tells you they can build the happy path. What tells you the rest:

• Show me the failure modes you've handled, and what the system does for each.
• Show me the eval report - pass rate and failure attribution.
• Show me where a human approves, and what they see when they do.
• Show me the trace of what the system did yesterday.

Good answers to those four questions are what "it works" actually means. If you're evaluating a system - yours, a vendor's, or something you're about to commission - those are the questions I'd start with, and it's a review I'm happy to do with you.