The signal thins out
Remember the game of telephone? A sentence whispered through seven people, and by the last one it has gone crooked. Deep networks hit exactly that problem.
Do you remember the game of telephone from childhood? A sentence whispered from ear to ear through five or seven people, and by the last one the content has gone almost entirely crooked. It turns out machine learning, once stacked into many layers, runs into exactly that trouble, and the field calls it the vanishing gradient.
Recall from the other day that a machine learns by looking at where the result went wrong and tracing back to adjust each layer. The problem is that this “correction notice” grows fainter with every layer it passes through, exactly like a sentence thinning out through each person in the game. With one calculation that was common for a time, the sigmoid function, the notice shrinks to under a quarter at each layer; travel back through ten layers and almost nothing remains. By the time it reaches the first layers, the ones that should be learning the most foundational things, there is barely any notice left to correct from.
The field took years to untie this knot, with new calculations and “shortcuts” connecting layers directly so the signal does not thin out. Because of them, in 2015 a model 152 layers deep, called ResNet, won decisively at the prestigious ImageNet image recognition competition.
I find this mirrors an ailment of organisations with many tiers. Intent passed down from above, or feedback passed up from below, thins at each intermediate level, arriving as a faint shadow of itself. In your organisation, how many tiers does an important signal pass through before reaching the person who needs to hear it?
To your desk
In a deep model the learning signal thins as it passes through layers, and by the furthest one it has all but vanished. Information in an organisation leaks the same way.
This week, try picking a recent important message and tracing how many levels it passed through before reaching the person doing the work, and how much of the original survived. Often the trouble is not the content but the number of tiers it had to cross.
A direct line of exchange, at the right moment, usually preserves more than a winding multi-step process.