Let us assume a 1-dimensional regression problem (with one input variable). We are given a set of data points that are NOT collinear and our objective is to fit a straight line that best fits the given data points. So basically the hypothesis function would be of the form h(x) = theta0 + theta1 * x where theta0 and theta1 are the model parameters and x is the input variable. This linear regression problem aims to determine the optimal values of theta0 and theta1 that best fit the given data points by minimizing the mean squared error cost function. For this specific scenario, does adding regularization term to the cost function help? If so, please explain how it helps?
The parameters that minimize the SSE or MSE cost function will be whatever the OLS matrix algebra give. If you use regularization, you will not get as small of an SSE or MSE value when you put your data through the regression equation.
So regularization is worthless, right? WRONG!
If you're doing regularization, than you're probably most interested in predictive ability. This means that you will want to test your data out-of-sample. Briefly, if you have 100 observations, train your model on 80. Then try it out on the remaining 20. If you get an acceptable SSE or MSE on those 20 points, then you have a good predictive model. If you kick butt on the 80 points but have miserable performance on the 20, your model has poor predictive performance. Regularization helps in this situation. Sure, you are penalized on the 80 in-sample points, but you may have superior performance on the 20 out-of-sample points, which is where it really counts.
I think it depends on what "help" means. Regularization tends to decrease the variance of the estimators in exchange for some increase in bias. If your model is well specified and there is only one predictor - there is no multicoliniarity by definition, and so if you've a reason to believe that the regression assumptions hold then adding a regularization term will only make things worse because it will increase the bias of the estimator and you won't gain much by reduction in variance because that wasn't a problem in the first place. Remember that under the assumptions I have mentioned the simple OLS estimator is BLUE (Best Linear Unbiased Estimator) by Gauss-Markov theorem.
Adding the regularization term will make the regression line less vulnerable to outliers. In other words, it'll help your regression line focus more on finding the underlying pattern than in fitting all the data points to the line. But in the setting, you mentioned above when all data points are collinear it will be of no help.