4
$\begingroup$

According to the paper https://arxiv.org/pdf/1502.03167.pdf,

It has been long known (LeCun et al., 1998b; Wiesler & Ney, 2011) that the network training converges faster if its inputs are whitened – i.e., linearly transformed to have zero means and unit variances, and decorrelated.

My question is why would a network learn better from uncorrelated inputs?

My intuition for this is that if your inputs (X,Y) are independent (and in particular uncorrelated), you would expect the same conditional distribution of X no matter the value of Y, thus the data is in some sense more regular. But this is a very handwavy intuition and I would like to understand this claim on a deeper level.

Improve this question
$\endgroup$
3
  • $\begingroup$ what do you mean by decorrelated inputs? $\endgroup$
    – rep_ho
    Oct 8, 2022 at 11:53
  • $\begingroup$ @rep_ho I am using the paper's terminology but it just means 'uncorrelated'. I guess the 'de' part implies that we somehow transform the inputs so that they are uncorrelated (as opposed to the inputs being given to us this way). Edited for clarity $\endgroup$
    – mich
    Oct 8, 2022 at 12:01
  • 2
    $\begingroup$ The quotation text in parenthesis contains citations to papers that discuss this topic in more detail. $\endgroup$
    – Sycorax ♦
    Oct 8, 2022 at 17:21

1 Answer 1

Reset to default
8
$\begingroup$

Basically, gradient descent works best when the error surface is spherical - the same curvature (ie gradient of gradient) in each direction.

This is because you want to adjust the step size according to the curvature. Imagine in 1-d, if you have very strong curvature you need to take very small steps (or you will overshoot). If you have very low curvature you need a very large step size, or you don't get anywhere. Now in more dimensions, different directions will have different curvature, and you will be forced to use the smallest step size across all your directions or you will overshoot the minimum, but that will mean you traverse other directions very slowly.

Now, if you consider the error surface of linear regression (ie take the 2nd derivative of the mean squared error wrt the weights(coefficients), you will see that it is actually $X^TX$. Whitening the inputs ($X$) will therefore give you a nice, spherical, bowl shape. So to the extent that your problem has a linear component, you might expect this behaviour to carry over to your nonlinear problem.

see eg https://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/slides/lec07.pdf around slide 19

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your answer. What I don't understand is why should we expect our problem to have a linear component? In a NN your input is undergoing a lot of nonlinear transformations, leaving no linear part remaining... $\endgroup$
    – mich
    Oct 16, 2022 at 0:45
  • $\begingroup$ And maybe a more technical question - in linear regression there should be a row of all 1s in the $X$ input matrix (representing constants), so wouldn't this mean X^TX is never going to be a "nice, spherical bowl shape", even if the actual input data is whitened? $\endgroup$
    – mich
    Oct 16, 2022 at 0:49
  • $\begingroup$ I would argue that the success of residual networks is precisely to allow linear problems to be solved linearly. ( a linear multilayer neural network has nonconvex error surface). $\endgroup$
    – seanv507
    Oct 17, 2022 at 13:51
  • $\begingroup$ if X is whitened and X_b is the enlarged design matrix with added column, b, of all 1's, then b.b = 1 and x.b =0 (because each column x of X has zero mean) and x.x = 1 and x1.x2 = 0 (for different columns x1, x2 of X), so X_b^T X_b is the identity matrix giving you the bowl shape. $\endgroup$
    – seanv507
    Oct 17, 2022 at 13:57
  • $\begingroup$ I am certainly not claiming this is a proof. even for logistic regression we get stats.stackexchange.com/questions/68391/…. $\endgroup$
    – seanv507
    Oct 17, 2022 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.