2
$\begingroup$

What is it mean by the statement "X and Y explained a greater proportion of variance in Z than X alone "

More generally, what is it meant when the model "explains more variation" in the dependent variable -- certainly this is not equivalent to "this explains the variable" more?

Thank you for the insight into this basic question.

$\endgroup$
  • $\begingroup$ "Explaining variance" is just as vague as "explaining a variable" to me, so I wouldn't exclude the possibility that they're (semantically) related. I do not prefer this way of interpreting ANOVA/regression output because it's misleading and "unuseful" information. Suppose I said "exercise explains variance in blood pressure".. does that mean if I exercise my blood pressure will become less variable? In fact, my BP becomes more variable since bouts increase my BP and my resting pressure will tend to normotensive which is desirable. There are better ways to describe results. $\endgroup$ – AdamO Aug 6 at 20:45
  • $\begingroup$ I have replaced the generalized-linear-model (glm) tag with multiple-regression. The question could conceivably be generalized to glms by reference to deviances rather variances and sums of squares, but that does not appear to be the OP's intention. $\endgroup$ – Gordon Smyth Aug 16 at 6:57
2
$\begingroup$

In a couple of words (the shorter the better!) when you add a variable to a model, if the added variable adds some explanatory power, then the addition increases the model fit (i.e. the capacity of the model as a whole of predicting the dependent variable in the sample where the model is estimated). However, bear in mind that adding more variables also entails a higher risk of overfitting (i.e. building a model with a high fit within the sample over in which it is estimated and a degraded prediction performance when used on other samples). So over time some specification criteria have been introduced such that they balance the number of parameters to be estimated against the model fit, so that the addition of variables (and therefore parameters to be estimated) may be discouraged when the resulting increase in the mode fit is not high enough compared to the parameter penalization.

With regard to your question "More generally, what is it meant when the model "explains more variation" in the dependent variable -- certainly this is not equivalent to "this explains the variable" more?" in basic models like regression, the more variance of the dependent variable is explained by the model, the less is explained by residuals, the better the model is because (to use your words) “it explains the dependent variable more”

$\endgroup$
5
$\begingroup$

We have to be thinking about a model to answer your question so let's assume a linear model. For convenience, we'll use sums of squared deviations instead of variances; to translate for variances, divide through the sums of squares by $N - 1$.

Let $Z = (z_1, ..., z_N)$ be your data; it has sum of squared deviations $\sum_{i = 1}^N (z_i - \bar{z})^2$. If you decide to estimate $Z$ as $\hat{Z} = \beta_0 + \beta_1 X + \beta_2Y + \varepsilon$, then you obtain estimates $\hat{Z} = (\hat{z}_1, ..., \hat{z}_N)$ for $Z$; its mean is the same as $Z$'s mean.

It is a fact that the sample variance of $\hat{Z}$ is less than that of $Z$, intuitively because we have constrained it to be on a line. Their variance is only the same if the data is exactly linear; therefore the idea is that by trying to capture $Z$ with this estimate, you are trying to capture the variation of $Z$. So the more variance $\hat{Z}$ captures, the closer the data is to being exactly linear.

The following identity holds (called the ANOVA decomposition):

$$\underbrace{\sum_{i = 1}^N (z_i - \bar{z})^2}_{\text{TSS}} = \underbrace{\sum_{i=1}^N (z_i - \hat{z}_i)^2}_{\text{RSS}} + \underbrace{\sum_{i=1}^N (\hat{z}_i - \bar{z})^2}_{ESS} $$

So the total sum of squares (TSS) of $Z$ breaks up into the explained sum of squares (ESS), which is the (unnormalized) variance of the fitted data. This is the "explained variance". The residual sum of squares (RSS) is how much the real data still differs from your fitted data---the "unexplained variance". To get a proportion of explained or unexplained variance, you can divide either by TSS. The proportion of explained variance, $ESS/TSS$ is called the $R^2$ value and measures the quality of fit.

The language of explained/unexplained variance isn't always useful; I really only see it with linear regression and PCA. Also, explaining as much variance as possible isn't the best idea if you want to do prediction, since this is overfitting. When you do something like ridge regression, you get a biased estimate which would "explain less variance"---the $R^2$ on the data will be worse---but the reason you do ridge regression is because the test error will usually be better.

(For prediction, more important than the ANOVA decomposition is the bias-variance decomposition.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.