When minimizing the squared error using vanilla/basic gradient descent..meaning basically:

while not converged:
   new_parameter = old_parameter - step_size*(gradient of rss)

Are the following assumptions correct?

  • You're guaranteed convergence to a global minimum because rss is a convex problem
  • The only reason it wouldn't converge is that you chose a step size that was too large
  • If you get convergence under a particular set of initial weights, w1, and a certain step size, s1, then the only reason you wouldn't get convergence with a slightly altered set of initial weights, w2, with the same step size, s1, is because the step size is no longer sufficient to bring about convergence with the altered weights?

In other words, what would cause a failure of convergence with a slightly altered set of weights? I'm assuming it's the step size. Are there any other reasons?

  • $\begingroup$ Is this question hypothetical or are you experiencing this in real data? $\endgroup$ – shadowtalker Jan 19 '17 at 18:43
  • $\begingroup$ RSS is convex for a problem like OLS with $p<n$ but RSS is not (strictly) convex in general. $\endgroup$ – Sycorax Jan 19 '17 at 22:39
  • $\begingroup$ well, okay fine. For this particular problem, it's actually strongly convex, and yes, it was experienced in real data (for someone) and I'm 99% sure the only reason they didn't reach convergence was due to a step size problem. I've never heard of choosing the wrong initial weights being an issue in a regression problem, other than perhaps problems with small datasets, hence the post to see what others had to say. $\endgroup$ – user127039 Jan 20 '17 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy