AI needs receipts
Why consequential software has to show its work before it earns trust.
Correct answers aren't enough. In AI product design, the harder ask is whether someone can see why the system thinks it's right, and what breaks if it isn't.
Teams love benchmark scores because they make the product feel objective. Scores don't help the person who has to approve a payroll adjustment, publish a customer policy, or explain a bad recommendation to a manager. In high-consequence work, people need orientation: where did this come from, what data carried the risk, what changes if I accept it.
I think of receipts as evidence surfaces. They don't have to be verbose. They just have to show up where the stakes are real.
A useful receipt usually has three parts: a trace of the inputs behind the answer, a clear preview of what will change if the action runs, and an undo path cheaper than paranoia. Users won't inspect every one. They need to know the product leaves room for verification when they want it.
Trust gets described as an emotion. In software it's usually structure. People trust systems that can show their work.
Once you design for receipts, the UI changes shape. Summaries become collapsible instead of final. Approvals become previews instead of permission dialogs. Logs stop feeling like admin exhaust and start feeling like part of the product. That's the shift I care about: AI that behaves less like a magician and more like an operator who can explain the call. The same problem shows up when automation acts without a visible mandate — see Consentful Automation.