I fit a regression model on a data set and get some in-sample RMSE. I wanted to know, how likely is that I get this good RMSE (or even better) under assumptions that there are no patterns in the data.
To answer this question, I take my real targets, resample from them, and replace the real targets by "fake" targets. By doing this I destroy any potential dependency on features.
To make it simple to understand, you can imagine that instead of a resampling I just randomly permute all the targets. So, obviously there is no dependency on features.
Now, I use my "fake" data set to fit the model again. I do it many times and what I see is that I never get results that are as good as on the real data set.
So, my interpretation is that the model fit some pattern (signal) on the real data set and therefore it is always better than on a fake (randomized) data set.
Now I do the same but with a gradual increase of the regularization parameter. What I see, is that difference between the performance in real and fake data sets gets smaller and smaller.
So, I assume that the regularization makes the model less sensitive to noise (which is good) but, at the same time, it makes the model less sensitive to signal (pattern).
So, now I come to my question: Can a regularization harm more than help in the situation of a huge over-fit?
What I mean by that, is that by a regularization model becomes less sensitive to noise but also it becomes much more insensitive to signal!