2
$\begingroup$

I fit a regression model on a data set and get some in-sample RMSE. I wanted to know, how likely is that I get this good RMSE (or even better) under assumptions that there are no patterns in the data.

To answer this question, I take my real targets, resample from them, and replace the real targets by "fake" targets. By doing this I destroy any potential dependency on features.

To make it simple to understand, you can imagine that instead of a resampling I just randomly permute all the targets. So, obviously there is no dependency on features.

Now, I use my "fake" data set to fit the model again. I do it many times and what I see is that I never get results that are as good as on the real data set.

So, my interpretation is that the model fit some pattern (signal) on the real data set and therefore it is always better than on a fake (randomized) data set.

Now I do the same but with a gradual increase of the regularization parameter. What I see, is that difference between the performance in real and fake data sets gets smaller and smaller.

So, I assume that the regularization makes the model less sensitive to noise (which is good) but, at the same time, it makes the model less sensitive to signal (pattern).

So, now I come to my question: Can a regularization harm more than help in the situation of a huge over-fit?

What I mean by that, is that by a regularization model becomes less sensitive to noise but also it becomes much more insensitive to signal!

$\endgroup$
4
  • 2
    $\begingroup$ within sample fit will always be larger for unregularized models. Regularization is there to improve out of sample predictions. Try to run your permutation test for LOOCV or 10foldCV rmse to see what happens $\endgroup$
    – rep_ho
    Aug 8, 2022 at 13:37
  • $\begingroup$ @rep_ho, yes, I know that regularization makes in-sample error larger (because of a reduction of over-fit). However, the results of my tests demonstrate that the more I regularize the model, the closer are the results on real and fake data sets! It means that the regularized models have more troubles to benefit from patterns present in the real data. Which is contra-intuitive to me and this is the reason why I ask the question. $\endgroup$
    – Roman
    Aug 8, 2022 at 15:07
  • $\begingroup$ What do you mean by "over-fit" in your question? Do you mean there is no relationship at all between the target variable and features but your model says there is a relationship? Or that there is a relationship but you've tuned your model too closely to the current data? $\endgroup$
    – Eli
    Aug 10, 2022 at 13:19
  • $\begingroup$ @Eli, the second one. I do have some patterns in the data but noise is huge and the model fitting not only the signal but also a large portion of noise. So, basically the model is to flexible, to expressive. $\endgroup$
    – Roman
    Aug 10, 2022 at 14:44

2 Answers 2

3
+25
$\begingroup$

If you increase the regularisation parameter too much, the model will ignore the data completely and the norm of the weights will be forced arbitrarily close to zero". This ought to be the solution you would asymptotically get modelling fake targets, as there is no correlation between the attributes and the targets. Making generalisation worse by regularising too much (or by using a model that is too structurally simple, e.g. linear regression for a non-linear problem) is known as "under-fitting".

The key to regularisation lies in tuning the regularisation parameter to optimise out-of-sample performance without over- or under-fitting. I tend to tune the regularisation to minimise the leave-one-out error (because it is cheap for many useful models) or use a Bayesian approach.

The regularisation parameter is equivalent to placing a bound on the magnitude of the weights (see my answer to this question). This creates a nested set of models of increasing complexity as the bound is made looser (by reducing the regularisation parameter). If you increase the bound slightly, the model can implement all of the functions it previously could, plus a few more. Thus the value of the regularisation parameter is an indication of the complexity of the model class that you are fitting to the data. In tuning the parameter, you are matching the complexity of the model to suit the complexity of the learning problem. You don't want it to be too complex, but you don't want it to be too simple either.

$\endgroup$
1
$\begingroup$

Yes! Regularization can destroy your model.

For example if we use ridge regression our loss is: $ \Sigma(y_{true,i} -y_{pred,i})^2 +\lambda a_j^2 $

Where $a_j$ are the regression coefficients.

If we take $\lambda$ is taken as 0, we get the optimal ols fit which may also fit to noise.

But if we take \lambda to infinity we force all our model coefficients to go to zero, forcing the model just to become $y=0$ and kill all the signal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.