When minimizing the squared error using vanilla/basic gradient descent..meaning basically:
while not converged: new_parameter = old_parameter - step_size*(gradient of rss)
Are the following assumptions correct?
- You're guaranteed convergence to a global minimum because rss is a convex problem
- The only reason it wouldn't converge is that you chose a step size that was too large
- If you get convergence under a particular set of initial weights, w1, and a certain step size, s1, then the only reason you wouldn't get convergence with a slightly altered set of initial weights, w2, with the same step size, s1, is because the step size is no longer sufficient to bring about convergence with the altered weights?
In other words, what would cause a failure of convergence with a slightly altered set of weights? I'm assuming it's the step size. Are there any other reasons?