Batch Normalization vs Full Whitening

Question

In section 3 of the paper Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, by Sergey Ioffe and Christian Szegedy, it says

Since the full whitening of each layer's inputs is costly and not everywhere differentiable, we make two necessary simplifications.

I understand that the full whitening is related to their analysis in section 2, however, I still couldn't figure out the difference between the full whitening and batch normalization. I think they are basically the same -- subtracting the mean from the inputs and then dividing by the variance. Where am I wrong?

Answer 1

Full whitening has an additional first step of projecting the data on eigven-vectors, which requires decomposition of a full covariance matrix computed for the data. For high-dimensional data, this can be very expensive. For additional info, see this link.