Confidently wrong

This is the risk I raise first whenever someone is about to hand important work to AI without checking it.

What hallucination is

Picture a charming storyteller with a firm voice, reeling off figures so smoothly that everyone believes them. The trouble is that a few of those details they invented, without being aware of it themselves. In AI this is called hallucination: the model produces information that sounds plausible and reads fluently, and is not true.

It happens because a language model is designed to guess the next word that sounds smoothest, not to look facts up. In 2023 there was a well-known case: two lawyers in New York were fined $5,000 by a court for filing a brief citing six cases entirely invented by ChatGPT, with case names and quotations that read exactly like the real thing.

How this applies to running a business

I find the lesson is not "do not use AI" but "do not skip the check". AI is well suited to drafting, to suggesting, to handling volume. But at the points where one wrong figure or one invented citation could do real damage, such as contracts, financial figures or legal advice, a person has to be the final check.

The healthy approach is to design a stopping point for checking into the process, especially for information whose accuracy can be verified. The more confident it sounds, the harder you should look, because it is precisely the fluency of AI that lulls its users.

In your process, where is AI output going straight out without a person checking it first?