The black box problem

Something troubles the AI field: a model can predict very accurately and still be unable to say why it concluded what it did. People call this the black box problem. We see the data go in, we see the result come out, and the middle is close to unreadable. The line of research looking for an answer is called explainable AI, shortened to XAI.

This is taken seriously enough that in 2017 a major American research agency, DARPA, set up a four-year programme bringing together eleven research groups to answer one question: how to make a model not only give an answer but explain the basis for it. The irony is that the more powerful the model, the harder it becomes to explain. And in consequential places like medicine, banking or hiring, a prediction without a reason is a prediction people find very hard to rest on.

In the project defences on my research programme, the question the panel repeated was not “how accurate is the model” but “who is accountable when it is wrong, and can we trace why it was wrong”. A trustworthy system is one willing to point at its own basis, and equally willing to say “I do not have enough evidence to conclude”.

The same holds for how an organisation decides. A decision the people carrying it out do not understand is very hard for them to genuinely stand behind, and harder still to correct when it drifts off course.

When an important decision is made in your organisation, do people understand why, or only that they must follow it?

To your desk

A trustworthy model is not one that is always right. It is one that can explain why it concluded what it did, and knows to stop when the basis is thin.

This week, try bringing that principle into your use of AI at work: each time you get an important answer, ask one more question, “based on what?”. If the tool cannot point to a source or a line of reasoning, treat it as a suggestion to consider rather than a conclusion to act on.

This small habit is also well worth teaching children as they start using AI: do not rush to believe, ask “how do you know this is right?”