Let us assume a 1-dimensional regression problem (with one input variable). We are given a set of data points that are NOT collinear and our objective is to fit a straight line that best fits the given data points. So basically the hypothesis function would be of the form h(x) = theta0 + theta1 * x where theta0 and theta1 are the model parameters and x is the input variable. This linear regression problem aims to determine the optimal values of theta0 and theta1 that best fit the given data points by minimizing the mean squared error cost function. For this specific scenario, does adding regularization term to the cost function help? If so, please explain how it helps?
$\begingroup$
$\endgroup$
5
-
$\begingroup$ Still helps with collinearity and overfitting. $\endgroup$– user2974951Commented Sep 9, 2019 at 8:01
-
1$\begingroup$ "Help" to do what? $\endgroup$– Peter FlomCommented Sep 9, 2019 at 10:34
-
$\begingroup$ I have made the question more clear and specific. Can you please take a look at it? $\endgroup$– Lakshmi SrinivasanCommented Sep 10, 2019 at 16:43
-
$\begingroup$ Your question is inherently contradictory: if you are modifying the "mean squared error cost function," then you are perforce not minimizing it! $\endgroup$– whuber ♦Commented Sep 10, 2019 at 20:04
-
$\begingroup$ @PeterFlom I have updated my question. Hope it is clear now. Can you please take a look at it? $\endgroup$– Lakshmi SrinivasanCommented Sep 12, 2019 at 4:18
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
I think it depends on what "help" means. Regularization tends to decrease the variance of the estimators in exchange for some increase in bias. If your model is well specified and there is only one predictor - there is no multicoliniarity by definition, and so if you've a reason to believe that the regression assumptions hold then adding a regularization term will only make things worse because it will increase the bias of the estimator and you won't gain much by reduction in variance because that wasn't a problem in the first place. Remember that under the assumptions I have mentioned the simple OLS estimator is BLUE (Best Linear Unbiased Estimator) by Gauss-Markov theorem.
$\begingroup$
$\endgroup$
1
The parameters that minimize the SSE or MSE cost function will be whatever the OLS matrix algebra give. If you use regularization, you will not get as small of an SSE or MSE value when you put your data through the regression equation.
So regularization is worthless, right? WRONG!
If you're doing regularization, than you're probably most interested in predictive ability. This means that you will want to test your data out-of-sample. Briefly, if you have 100 observations, train your model on 80. Then try it out on the remaining 20. If you get an acceptable SSE or MSE on those 20 points, then you have a good predictive model. If you kick butt on the 80 points but have miserable performance on the 20, your model has poor predictive performance. Regularization helps in this situation. Sure, you are penalized on the 80 in-sample points, but you may have superior performance on the 20 out-of-sample points, which is where it really counts.
-
$\begingroup$ Thanks for the vivid explanation! $\endgroup$ Commented Sep 13, 2019 at 5:24
$\begingroup$
$\endgroup$
7
Adding the regularization term will make the regression line less vulnerable to outliers. In other words, it'll help your regression line focus more on finding the underlying pattern than in fitting all the data points to the line. But in the setting, you mentioned above when all data points are collinear it will be of no help.
-
1$\begingroup$ I believe your answer might be incorrect, are you sure there will be "no effect"? I think that in the case the points are collinear, regularization would make the regression slope slightly smaller than what is in the data. $\endgroup$ Commented Sep 10, 2019 at 9:44
-
$\begingroup$ @MartinModrák Thank you for pointing out. What I actually meant to say was 'there will be no positive effect'. please correct me if I'm wrong $\endgroup$ Commented Sep 10, 2019 at 11:05
-
$\begingroup$ I didn't mean that the given data points are collinear. I have updated the question to make it more clear. $\endgroup$ Commented Sep 10, 2019 at 16:48
-
$\begingroup$ Based on the answer you provided above, I have a question. Only when you choose a non-linear hypothesis function (such as quadratic, cubic, ..) to fit the given data points, regularization helps by filtering the noise (outliers) in the data. But if the selected hypothesis function is linear, the regression problem is going to identify a straight line that best fits the data points. So how does regularization help in this scenario? $\endgroup$ Commented Sep 10, 2019 at 16:57
-
1$\begingroup$ Implicitly equating "no outliers" with "all collinear" is extremely confusing--how is collinearity related to being an outlier?--and doesn't seem to make any sense. Are you sure this is what you intended to write? $\endgroup$– whuber ♦Commented Sep 10, 2019 at 20:02