When the model drifts over time

This is a reminder I hold on to: right today does not mean right forever.

What model drift is

Picture a very accurate map of a city. A few years later, new roads have opened, old ones have changed direction, and that map, without having changed at all, misleads more and more. Model drift is exactly that: a model that was right at the time of training grows less accurate as the world around it changes, while it stands still on what it learned.

People distinguish two forms: the input data changes, called data drift, and the relationship between inputs and outcome changes, called concept drift. The second is far harder to detect. A classic example is Google Flu Trends, a tool that forecast flu outbreaks from search volume. According to the analysis by Lazer and colleagues published in Science in 2014, at the peak of the 2012 to 2013 flu season it predicted roughly twice the figure recorded by the US health authorities, and erred on the high side in 100 of 108 consecutive weeks.

How this applies to running a business

I find the lesson very clear: reliability has to be tracked continuously, not measured once and trusted forever. A demand forecasting model, a customer scoring rulebook, even a formula for success that once held, can all drift when the market, the customers or the competition change.

This holds for management assumptions too. A strategy that suited the market three years ago may be drifting now without our noticing, precisely because it once was right. The habit of periodically rechecking whether an assumption still holds is how drift is caught early, before it becomes a large mistake.

Which assumption, or formula that once worked, have you not rechecked lately against how things actually are today?