I have a multi-variable linear model. More precisely, a set of variables to explain a target variable. After adding Gaussian noise ($\sim N(0,1))$, the MSE increased from roughly $21$ to roughly $22$.

How can it be explained? Is that what's called overfitting?

  • $\begingroup$ To what do you add the noise? To the dependent variables in the training set? Or in the test set? Or to independent variables? If so, in which set? $\endgroup$ – Stephan Kolassa Jun 23 '17 at 9:53
  • $\begingroup$ I added the noise to the test set's y. I'm using the well-known data-set: cs.toronto.edu/~delve/data/boston/bostonDetail.html $\endgroup$ – Covvar Jun 23 '17 at 9:55
  • $\begingroup$ the dependent variable is medv $\endgroup$ – Covvar Jun 23 '17 at 9:58

If you add noise to the test set's dependent variables, then your predictions of the noisy (noisier) DVs will be worse, so your MSE will go up. After all, you cannot predict the random noise - that's why it is random noise.

Here is a thought experiment: suppose you have a perfect model and can predict the test set's DV perfectly. Your MSE is zero. Now you add noise to the test set's DVs. It doesn't make sense to change your predictions (since, as above, you can't predict your noise), so all that happens is that the DVs get perturbed. Before, you had perfect predictions. Now, they are not perfect any more. Your MSE increases from zero to some larger number.

And since the MSE is roughly the same as the variance of a normally distributed variable, your MSE should increase roughly by the variance of the noise you are adding.

  • $\begingroup$ After running it a few times, I can see that the MSE of the noisy DV can be lower as well, although most of times it's higher. $\endgroup$ – Covvar Jun 23 '17 at 10:30
  • $\begingroup$ Yes. There is always an element of randomness involved. It can also be larger than the noise variance, simply by chance. $\endgroup$ – Stephan Kolassa Jun 23 '17 at 10:31
  • $\begingroup$ could you explain the last sentence please? $\endgroup$ – Covvar Jun 23 '17 at 10:47
  • $\begingroup$ I'll try. Which sentence do you mean? In my answer, or in the later comment? Best to quote. $\endgroup$ – Stephan Kolassa Jun 23 '17 at 10:56
  • $\begingroup$ Why is the MSE is roughly the same as the variance of a normally distributed variable? $\endgroup$ – Covvar Jun 23 '17 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.