I read this Do deep nets really need to be deep? paper. There is one line that seems important but I don't understand. In section 2.3, it is told that

Because there are many highly correlated parameters in this large weight matrix, gradient descent converges slowly

My questions are:

  1. What it means with highly correlated parameters? What makes the paper said that the parameters are highly correlated?
  2. Is highly correlated parameters means the weight matrix has low rank?
  3. What is the connection between highly correlated parameters with the gradient descent converges slowly? Why with highly correlated parameters, the gradient descent converges slowly?
  • $\begingroup$ I would contact the authors (and post answer here!). There is a problem with highly correlated inputs (not parameters) - effectively gradient descent works best when the error surface is curved the same in each direction (then a single step size works well in any direction)..Just considering a single linear neuron ( ie linear regression by gradient descent), if you have correlated inputs then your error surface will be elliptical - changing much more along the principal axis... (the principal eigenvectors)...but now a single step size doesn't work well. $\endgroup$ – seanv507 Jan 19 '16 at 10:43

General problem with correlated inputs: In general, it is possible to have correlated input variables, which leads to correlated weights. Let's take an extreme example and lets assume you have a duplicate feature, $x_1 = x_2$ (perfect correlation), and you want a linear function that maps $X$ to $Y$, $Y = f(X)$, where $f(X) = \beta_0 + \beta_1x_1 + \beta_2x_2$.

Assume $\beta_1 = 0, \beta_2 = 1$ gives an answer that is "ok"; but so does $\beta_1 = 1, \beta_2 = 0$, and every $\beta_1 + \beta_2 = 1$ will give the same answer.

This means that changing $\beta_1$ has a significant effect on the value $\beta_2$ should take. As Neural Networks compute such a linear model at each node, it can happen a lot from a low correlation to begin with.

Neural Net specifics: There is also another problem, more specific to the structure of neural networks, that will lead to this. Again, the extreme exemple: if two nodes in the same layer are initialized with the same values, they will have the same gradient at each iteration and learn the same transformed feature out of the last layer, leading to a duplicate.

Effect on gradient descent: This argument might be a bit loose, but Gradient Descent works 'best' when the direction of the gradient at each iteration points to the optimal point; that is, you could minimize each $\beta_i$ separately and get to a good answer. This is possible when the function to optimize is stricly convex. But when inputs are highly correlated, this is no longer the case. Obviously, it is not possible for neural networks since the function is not convex to begin with, but it has effects on reaching the local minimum as well.

A little bit off-topic, but to help figure out what is going on in gradient descent, you can check lecture 6.2 of Geoffrey Hinton's mooc; "A Bag of tricks for gradient descent" (Here on Coursera, Here on Youtube). He is not stricly talking about correlation, but he shows the effect of scaling and shifting the inputs, which helps get an intuition about how gradient descent works.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.