In the linear regression problem, using a simple linear model with 1 variable & with 2 model parameters, performing batch Gradient Descent(GD) & assuming I am using Mean Square error as my cost function, will ensure that we will get only 1 local minimum which is global minimum!. So,the GD will always converge keeping the learning rate to neither too small or too large!

cost function surface

  1. But if we increase the number of model parameters lets say to 5 what will happen in the above setting?

  2. Also, when we just keep simple linear model and make the feature in higher order polynomial or take sin/ cos of the feature! what will happen?

  3. In the above case if we increase the number of variables, with high order polynomials or taking sin, then what will happen?

Could anyone help me understand these 3 serious, if you could add 3-D plots of these functions or even contour plots(additionally, if you could argue simultaneous how the behavior of batch gradient descent will change in all 3 scenarios), it will really help to get a clear understanding of each of these scenarios!

NOTE: I asked for plots just for an understanding of these 3 scenarios with 2 parameters & later an explanation(connecting intuition) for >3.

  • $\begingroup$ Generally speaking, as the error surface becomes increasingly complex the number of minima increases such that the ere are many local minima. In your 3D surface plot, there is a single local minimum which is also the global minimum - with increasing model complexity this will not be the case. $\endgroup$ – James Phillips Dec 12 '18 at 9:56
  • 2
    $\begingroup$ It sounds like you're asking under what conditions the quadratic cost function is strictly convex. Is that your question, or are you asking something else? $\endgroup$ – Sycorax Dec 12 '18 at 21:22
  • 1
    $\begingroup$ Possible duplicate of Is regression line (for a simple "Y given X" regression, no interaction etc) always unique? $\endgroup$ – Sycorax Dec 12 '18 at 22:03
  • 1
    $\begingroup$ I don't know what you mean by "behave the same" or "will happen," but I do know--because you are minimizing a sum of squares in every case--that any portrayal of the objective function is going to look qualitatively just like your plot. $\endgroup$ – whuber Dec 14 '18 at 14:43
  • 1
    $\begingroup$ It is impossible to construct an algorithm that will find a global minimum for an arbitrary function.. However, using defined ranges, there are routines for finding global minima. Moreover, without also characterizing what the appropriate models are, the appropriate data transformations, and the appropriate norm(s) for minimization, there is no correct regression procedure. $\endgroup$ – Carl Dec 15 '18 at 20:31