Finding where you break, first

This is a habit I learned: to find out how solid something is, try breaking it yourself before anyone outside gets the chance.

What red teaming is

Picture someone hiring a locksmith, before handing the keys of a new house to a buyer, to try prying every lock and working every hinge, looking for the weak point before someone with bad intentions finds it. It sounds backwards, and it is the surest way to learn how solid the house really is. In AI that practice is called red teaming: deliberately attacking, misleading and baiting the model to expose where it breaks, then patching it before it goes into real use.

The scale of this work is already large. At the DEF CON 31 security conference in August 2023, the Generative Red Team challenge was the largest public exercise of its kind at the time, with more than 2,200 people exchanging over 165,000 messages probing the leading models for flaws, with backing from the White House.

How this applies to running a business

I find the spirit of it well worth bringing into management, not only for AI. Before launching a product, a pricing policy or a new process, deliberately asking "where will this break, who could exploit it, what is the worst case" surfaces the holes while fixing them is still cheap.

What matters is creating an atmosphere where people on the team dare to point at the weak spot without fearing blame. An organisation where everyone only says "it is fine" is usually one that does not yet know where it is about to break.

When did you last sit down deliberately to find where one of your important plans could break?