In the linear regression problem, using a simple linear model with 1 variable & with 2 model parameters, performing batch Gradient Descent(GD) & assuming I am using Mean Square error as my cost function, will ensure that we will get only 1 local minimum which is global minimum!. So,the GD will always converge keeping the learning rate to neither too small or too large!
But if we increase the number of model parameters lets say to 5 what will happen in the above setting?
Also, when we just keep simple linear model and make the feature in higher order polynomial or take sin/ cos of the feature! what will happen?
In the above case if we increase the number of variables, with high order polynomials or taking sin, then what will happen?
Could anyone help me understand these 3 serious, if you could add 3-D plots of these functions or even contour plots(additionally, if you could argue simultaneous how the behavior of batch gradient descent will change in all 3 scenarios), it will really help to get a clear understanding of each of these scenarios!
NOTE: I asked for plots just for an understanding of these 3 scenarios with 2 parameters & later an explanation(connecting intuition) for >3.