6
$\begingroup$

It seems to odd to me that we measure the explanatory power of a regression model in "percent of variance explained", or $R^2 = {\rm cor}(\hat{y},y)^2 = r^2$ even though we all know that variance is just an auxiliary quantity to compute the more meaningful measure of uncertainty which is the standard deviation. Risk in finance or uncertainty in prediction is measured by $\sigma$, not by $\sigma^2$. Knowing the reduction in variance in a regression model seems much less useful than the reduction in standard deviation.

In fact, whenever I try to explain $R^2$ to my students, I usually start by comparing the overall variation of y (as measured by $\sigma_y$) to the remaining variation around the regression line (measured by $\sigma_{\epsilon}$). That idea is adapted much more naturally than the comparison of the variances which really have no direct interpretation!

So I propose a new measure which is truly "the amount of standard deviation explained". We can quickly derive it: $$ R^2 (=r^2) = 1 - \frac{RSS}{TSS} \Leftrightarrow 1 - \frac{\sqrt{RSS}}{\sqrt{TSS}} = 1-\sqrt{1-r^2} $$ where RSS = "residual sum of squares" ($\approx \sigma_{\epsilon}^2$) and TSS = "total sum of squares" ($\approx \sigma_{y}^2$)

Comparing the traditional $r^2$ with the new measure $1-\sqrt{1-r^2}$ reveals that substantially stronger correlations $cor(\hat{y},y)$ are needed to result in similar "uncertainty reduction". For example, what one used to call a high value of $R^2 = 0.8$ explaining 80% of the variance, would have reduced the true uncertainty by merely 55%!

The graph below shows the stronger convexity of this alternative measure.

enter image description here

My question is twofold

  • What is the (historic?) reason that $R^2$—despite its shortcomings—has established itself as the main measure of uncertainty reduction reported in virtually every statistical software and textbook?

  • From a didactic perspective: what is a convincing way of explaining $R^2$ to (non stats majors) students in an introductory course on statistics? Would the "reduction in standard deviation"—as outlined above—not be easier to teach?

$\endgroup$
  • 1
    $\begingroup$ I dont understand, what is the question? $\endgroup$ – Repmat May 16 '16 at 14:45
  • 3
    $\begingroup$ Why standard deviation rather than mean absolute deviation? I perceive mean absolute deviation as much more natural, while standard deviation is quite difficult to perceive (just look at the formula...). A loss function based on mean absolute deviation arises more naturally in trading, but would you have any example of a loss function based on standard deviation? $\endgroup$ – Richard Hardy May 20 '16 at 19:33
  • $\begingroup$ @RichardHardy see stats.stackexchange.com/questions/81986/… $\endgroup$ – flies Nov 22 '16 at 22:28
  • $\begingroup$ @flies, thanks, I am aware of that thread. Is there anything in particular in there that you wanted to attract my attention to? $\endgroup$ – Richard Hardy Nov 23 '16 at 6:28
  • $\begingroup$ no, I just thought it was a relevant discussion $\endgroup$ – flies Nov 28 '16 at 19:37
2
$\begingroup$

There is nothing inherently wrong with measuring goodness-of-fit in a regression model with other monotonic transforms of the coefficient of determination, and indeed, there are several natural transformations that have useful interpretations. Before getting to these, it is useful to note the desirable mathematical properties of the coefficient of determination, and how it arises. The best way to understand this is to look an linear regression from a geometric perspective, where we consider the response and explanatory variables as vectors in Euclidean space.


Geometric analysis of the linear regression: Using OLS estimation in linear regression, the response vector is projected onto the column-space of the design matrix, which yields a predicted response vector in that column-space, and a residual vector that is perpendicular to that column-space (i.e., in the null space of the design matrix). This gives the regression equation $(\mathbf{y} - \bar{\mathbf{y}}) = (\hat{\mathbf{y}} - \bar{\mathbf{y}}) + \mathbf{r}$ which decomposes the response deviation into a predicted deviation and a residual. Since the residual vector and the predicted response vector are orthogonal under OLS estimation, this gives you a triangle of vectors that obey Pythagoras theorem, giving the decomposition:

$$\underbrace{\|\mathbf{y} - \bar{\mathbf{y}}\|^2}_{SS_\text{Tot}} = \underbrace{\|\hat{\mathbf{y}} - \bar{\mathbf{y}}\|^2}_{SS_\text{Reg}} + \underbrace{\|\hat{\mathbf{y}} - \bar{\mathbf{y}}\|^2}_{SS_\text{Reg}}.$$

Geometric explanations of linear regression commonly show the triangle of vectors in the above decomposition, and note the resulting decomposition of the squared-norms of these vectors (i.e., the sums-of-squares). One natural way to look at the goodness-of-fit of the linear regression model is to look at the shape of this triangular decomposition --- the smaller the residual vector (relative to the other vectors), the narrower the triangle and the better the fit of the model to the data.


Some useful measures of goodness-of-fit: There are several useful ways of looking at the goodness-of-fit, via the above decomposition of vectors (and the resulting decomposition of their squared-norms). Most of these involve using a measure that is some monotonic transformation of the coefficient of determination, and so they all measure the same thing, but in a different way. Here are some useful measures of the goodness-of-fit that are monotonic functions of the coefficient-of-determination:

  • Coefficient of determination: This statistic is defined as:$$R^2 \equiv 1 - \frac{SS_\text{Res}}{SS_\text{Tot}} = \frac{SS_\text{Reg}}{SS_\text{Tot}}.$$ As you point out, it measures the proportion of the variance that is "explained" by the explanatory variables. Geometrically, this statistic shows the proportionate size of the squared-norm of the predicted deviation, relative to the total deviation. The statistic is useful because it has a simple interpretation, and it has a simple geometric intuition.$$\text{ }$$

  • Regression vector angle: An associated measure of the goodness-of-fit is the angle $0 \leqslant \theta \leqslant \pi / 2$ between the response deviation vector and the residual deviation vector. Using standard trigonometric rules, this vector angle is obtained by the formula: $$\cos(\theta) = \frac{SS_\text{Reg}}{SS_\text{Tot}} = R^2.$$ Thus, it is easily seen that this vector angle is a monotonic transform of the coefficient-of-determination (a higher value for the coefficient-of-determination gives a smaller vector angle). This statistic is useful because it has a simple geometric interpretation --- a small vector angle means that the observed response deviation (from the mean) is close to the predicted response deviation (from the mean).$$\text{ }$$

  • F-statistic: Taking the number of data points and parameters as fixed, we can write the F-statistic for the goodness-of-fit of the regression in terms of the coefficient-of-determination:$$F = \frac{MS_\text{Reg}}{MS_\text{Res}} = \frac{n-m-1}{m} \cdots \frac{R^2}{1-R^2}.$$ Taking the number of data points and parameters as fixed, this statistic is a monotonic transform of the coefficient-of-determination. This statistic is useful because it has a well-known distribution under the null hypothesis that the true regression coefficients are all zero. It can therefore be used for hypothesis testing, to see if there is evidence of an association between the set of explanatory variables and the response.$$\text{ }$$

  • Explained standard deviation: Another useful measure of goodness-of-fit is the proportion of explained standard deviation, comparing the estimated standard deviation of the error terms and the response variables. This is given by: $$W \equiv 1 - \frac{\hat{\sigma}_\varepsilon}{\hat{\sigma}_Y} = 1 - \sqrt{{\frac{MS_\text{Res}}{MS_\text{Tot}}}} = 1 - \sqrt{\frac{n-1}{n-m-1} \cdot (1-R^2)}.$$ Taking the number of data points and parameters as fixed, this statistic is a monotonic transform of the coefficient-of-determination. It is similar to the statistic you propose to use, except that yours does not adjust properly for the degrees-of-freedom in the regression.


As you can see, there are several monotonic variations on the coefficient-of-determination that can be used as measures of goodness-of-fit. These have different interpretations that are all quite useful in certain contexts. The statistic you propose is somewhat similar to the explained standard deviation statistic above, but it fails to adjust properly for the degrees-of-freedom in the regression, so its interpretation would be a bit more difficult. Nevertheless, it is a monotonic function of the coefficient-of-determination, so it can be used as a measure of goodness-of-fit.

$\endgroup$
1
$\begingroup$

Particularly for a class for non-majors, it's probably fine. Most likely your students think of standard deviation whenever you mention variance, anyway.

It also makes sense to me. The goal of regression is to reduce the variability. Sure, the variability of human height is quite large, but when you account for age and gender, the distribution narrows, and you have a better idea of how tall a 30-year-old woman is ("around 5'5" ish") than how tall a human is in general.

So if you're reducing the variance of the conditional response distribution, you're also reducing the standard deviation of the response distribution. Where I am not clear is when it comes to quantifying that reduction, since the square root of an unbiased estimator of variance is a biased estimator of standard deviation.

Answering explicitly:

1) Variance is a natural way of thinking about the spread of data, since it's the second central moment of a distribution. There's also a good estimator that is unbiased for every distribution with a defined variance. Taking the square root of that (amazingly) results in a biased estimate of standard deviation.

2) If you're reducing the variance, you're also reducing the standard deviation. I'm unsure about the math of everything, since there's that nasty issue of unbiased variance becomes biased standard deviation, but that level of detail wouldn't make it into the class for non-majors.

$\endgroup$
1
$\begingroup$

If $X_1,\ldots,X_n$ are independent random variables then $$ \operatorname{var}(X_1+\cdots+X_n) = \operatorname{var}(X_1)+\cdots+\operatorname{var}(X_n). $$ Other measures of dispersion, including standard deviation and mean absolute deviation, lack this property.

If follows that \begin{align} & \Big(\text{explained part of the variance} \Big) \\ & {} + \Big( \text{unexplained part of the variance} \Big) \\[6pt] = {} & \Big( \text{total variance} \Big). \end{align} If not for that, would it make sense to speak of what percent of the dispersion is explained or unexplained?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.