Batch NormalizationBatch Normalization

In 2015, Sergey Ioffe and Christian Szegedy at Google introduced batch normalization, a technique that would appear in almost every serious deep network within a year. The problem it targeted was that as a network trains, the distribution of inputs to each layer keeps shifting, forcing every layer to chase a moving target.

Their fix was to normalize the inputs to each layer across the current mini-batch, rescaling them to a stable mean and variance, then letting the network learn how much to stretch and shift that normalized signal. Applied at every layer, it kept the internal numbers well-behaved.

The effect was dramatic. Networks trained faster, tolerated higher learning rates, and were far less sensitive to how their weights were initialized. Depths that had been painful to train suddenly became routine.

Ioffe and Szegedy attributed the gains to reducing what they called internal covariate shift. Later researchers challenged that explanation, arguing the real benefit is that batch normalization smooths the optimization landscape, making gradients more predictable. The debate is not fully settled.

Whatever the mechanism, it worked, and it spawned a family of relatives, layer normalization, group normalization, and others, tuned for cases where batch statistics are awkward, such as the transformers behind modern language models.

Batch normalization is a rare case where engineering ran ahead of theory: a method used billions of times a day whose exact reason for working is still argued over in papers.