4
$\begingroup$

I am trying to prep myself for data science interview & I saw the following question on a forum

In the case of least square regression, how does the variance of the error term change with the number of predictors?

I am not sure how to answer this? Any ideas?

Any help will be appreciated

$\endgroup$
1
  • $\begingroup$ You really need to make clear whether the "error term" refers to the noise in the data-generating process - the vector of epsilons in $Y = X\beta + \epsilon$ - or the residuals you find from the difference between your observed and fitted responses, $e = Y - \hat{Y}$. "Error term" usually refers to the former. The answers so far consider the residuals (I suspect the original question did too, but it is not well specified). $\endgroup$
    – Silverfish
    Jan 1, 2015 at 19:30

2 Answers 2

2
$\begingroup$

The variance of the error term decreases (or, at worst, does not increase) when you add more regressors. The reason is that a new variable can explain some more variability in the data that wasn't explained by previous regressors. This will reduce the unexplained variations in the data, whch will cause the variance of the error term to decrease.

$\endgroup$
4
  • $\begingroup$ "a new variable can explain some more variability" - this is not true, unless you put "explain" in quotes or something. You can throw in any variable, even completely irrelevant and pointless variable, yet it will most likely increase $R^2$. $\endgroup$
    – Aksakal
    Jan 1, 2015 at 18:11
  • $\begingroup$ @YairDaon Why does the (estimated?) variance of the error term decrease when one adds more regressors? See stats.stackexchange.com/questions/265827/… $\endgroup$
    – Elias
    Mar 8, 2017 at 11:35
  • $\begingroup$ @Monir Look if you first downvote my answer and then ask for clarification, don't expect me to take the time and answer you. $\endgroup$
    – Yair Daon
    Mar 11, 2017 at 1:34
  • $\begingroup$ @YairDaon Well, I downvoted after I commented (as I remember it) because I felt confident that your answer was either false or not well motivated. Before I upvoted your answer, so it is just 1-1=0. But now my vote is locked, sorry! In any case: What do you mathematically mean by "The variance of the error term decreases (or, at worst, does not increase) when you add more regressors."? $\endgroup$
    – Elias
    Mar 11, 2017 at 7:44
0
$\begingroup$

It doesn't increase. See these examples. Also, look at adjusted $R^2$, which doesn't have this defficiency.

Here's the MATLAB code example to show how adding a completely irrelevant and inexplanatory variable increases ordinary $R^2$, but decreases Adjsuted $R^2$.

The true process is $y=1+t+\varepsilon$, where $\varepsilon\sim\mathcal{N}(0,1)$. I estimate the true model, then estimate the model with an extra variable $y=1+t+z+\varepsilon$, where $z\sim\mathcal{N}(0,1)$, basically a noise. Yet, this garbage variable inflates ordinary $R^2$ goodness-of-fit metric!

rng(1)
X=(1:100)'/50;
beta = [1 1]';
y=X+ones(100,1)+randn(100,1);
mdl = fitlm(X,y);
disp 'Correct model'
mdl.Rsquared

X = [X rand(100,1)];
mdl = fitlm(X,y);
disp 'Extra variable model'
mdl.Rsquared

OUTPUT

Correct model

ans = 

    Ordinary: 0.2926
    Adjusted: 0.2854

Extra variable model

ans = 

    Ordinary: 0.2936
    Adjusted: 0.2790
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.