0
$\begingroup$

I have a multi-variable linear model. More precisely, a set of variables to explain a target variable. After adding Gaussian noise ($\sim N(0,1))$, the MSE increased from roughly $21$ to roughly $22$.

How can it be explained? Is that what's called overfitting?

Improve this question
$\endgroup$
3
  • $\begingroup$ To what do you add the noise? To the dependent variables in the training set? Or in the test set? Or to independent variables? If so, in which set? $\endgroup$
    – Stephan Kolassa
    Commented Jun 23, 2017 at 9:53
  • $\begingroup$ I added the noise to the test set's y. I'm using the well-known data-set: cs.toronto.edu/~delve/data/boston/bostonDetail.html $\endgroup$
    – Covvar
    Commented Jun 23, 2017 at 9:55
  • $\begingroup$ the dependent variable is medv $\endgroup$
    – Covvar
    Commented Jun 23, 2017 at 9:58

1 Answer 1

Reset to default
2
$\begingroup$

If you add noise to the test set's dependent variables, then your predictions of the noisy (noisier) DVs will be worse, so your MSE will go up. After all, you cannot predict the random noise - that's why it is random noise.

Here is a thought experiment: suppose you have a perfect model and can predict the test set's DV perfectly. Your MSE is zero. Now you add noise to the test set's DVs. It doesn't make sense to change your predictions (since, as above, you can't predict your noise), so all that happens is that the DVs get perturbed. Before, you had perfect predictions. Now, they are not perfect any more. Your MSE increases from zero to some larger number.

And since the MSE is roughly the same as the variance of a normally distributed variable, your MSE should increase roughly by the variance of the noise you are adding.

Improve this answer
$\endgroup$
6
  • $\begingroup$ After running it a few times, I can see that the MSE of the noisy DV can be lower as well, although most of times it's higher. $\endgroup$
    – Covvar
    Commented Jun 23, 2017 at 10:30
  • $\begingroup$ Yes. There is always an element of randomness involved. It can also be larger than the noise variance, simply by chance. $\endgroup$
    – Stephan Kolassa
    Commented Jun 23, 2017 at 10:31
  • $\begingroup$ could you explain the last sentence please? $\endgroup$
    – Covvar
    Commented Jun 23, 2017 at 10:47
  • $\begingroup$ I'll try. Which sentence do you mean? In my answer, or in the later comment? Best to quote. $\endgroup$
    – Stephan Kolassa
    Commented Jun 23, 2017 at 10:56
  • $\begingroup$ Why is the MSE is roughly the same as the variance of a normally distributed variable? $\endgroup$
    – Covvar
    Commented Jun 23, 2017 at 11:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.