Normalization techniques — stabilizing neural networks

During training, BatchNorm uses batch statistics. But at inference time, batch statistics may not be representative of the full dataset. Instead, BatchNorm maintains exponential moving averages (EMA) of mean and variance computed from all training batches. At inference, these running statistics are used instead.

The update rule is:
running_mean = (1 - momentum) × running_mean + momentum × batch_mean

A typical momentum value is 0.1, meaning each new batch contributes 10% to the running average. This makes BatchNorm have different behavior at train vs. test time, which can sometimes cause subtle bugs if not handled carefully.

After normalizing to zero mean and unit variance, normalization applies a learnable affine transformation: y = γx + β. These parameters (gamma and beta) allow the network to undo the normalization if that's optimal for the task.

For example, if a layer naturally benefits from non-unit variance activations, the network can learn γ ≠ 1 to scale back up. Similarly, β lets the network reintroduce a non-zero mean if useful. This flexibility is crucial—normalization should stabilize but not constrain the network unnecessarily.

CNNs + BatchNorm: CNNs typically have large batch sizes (32-256) and benefit from BatchNorm's per-feature normalization across the spatial dimensions and batch. The batch dimension provides stable statistics.

Transformers + LayerNorm: Transformers often have smaller batch sizes and variable sequence lengths. LayerNorm computes statistics per sample across the feature dimension, making it independent of batch size. This stability is critical for the attention mechanism, which is sensitive to input scaling.

Modern practice: Some models mix both—e.g., applying LayerNorm before attention and MLPs, but occasionally using GroupNorm or BatchNorm in certain fusion layers. The key is matching normalization to your data and compute budget.

Without normalization, activations can grow large or shrink small unpredictably. This causes gradients to be very large or very small, forcing you to use tiny learning rates (≤0.0001) to avoid divergence. With normalization stabilizing activations, gradients remain in a reasonable range, so you can use much larger learning rates (0.001-0.1).

This is why normalization is so transformative: it enables 5-10× higher learning rates, which directly translates to 5-10× faster training. The relationship is roughly:
effective_learning_rate ∝ 1 / mean_activation_magnitude

Normalization keeps activation magnitudes stable, so you can crank up the learning rate without fear.

Normalization adds a small computational overhead—computing mean and variance requires a full pass over the activations. However, this cost is usually 5-10% of the total training time and is more than offset by the 5-10× speedup from faster convergence and higher learning rates.

Inference time cost: In BatchNorm, using running statistics at test time has near-zero cost (just a subtract and divide). LayerNorm, RMSNorm, and GroupNorm also have minimal inference overhead. So normalization doesn't slow down inference significantly.

Normalization has a subtle regularization effect. By constraining activations to have fixed statistics, it reduces the degrees of freedom available to overfit. Some research suggests normalization acts as an implicit regularizer, reducing overfitting even without explicit L2 regularization.

However, normalization is not a substitute for explicit regularization (dropout, weight decay, etc.). The best practice is to combine normalization with other techniques. Modern networks often use: normalization + dropout + weight decay + data augmentation for best generalization.