3
$\begingroup$

In Rosenthal and Rubin (1979) ``A Note on Percent Variance Explained as A Measure of the Importance of Effects'', they give an example of where $r^2$ is deceptively low:

Suppose half the patients in a medical study are randomly assigned to a new medical treatment (X = 1) while the other half are assigned the standard medical treatment (X = 0). The dependent variable is “alive one year after treatment” (Y = 1) vs. “dead” (Y = 0). Suppose the study obtained the frequencies displayed in Table 1. Under the standard medical treatment, 30%of the patients live; while under the new treatment, 70% of the patients live. This certainly is a dramatic and important effect. Yet, for these data r2 = .16. The conclusion that the treatment is unimportant because it accounts for only 16%of the variance is simply wrong. Percent variance explainied can, in some cases, then, be a very deceptive measure.

The numbers are

      | Dead (Y=0) | Alive (Y=1) |
-------------------------------------
X = 0 | 35         | 15          | 50   
X = 1 | 15         | 35          | 50
-------------------------------------
      | 50         | 50          | 100

How do you actually calculate the $r^2$ here?

$\endgroup$
  • $\begingroup$ As a side note, it is common to think about a measure of effect size in these cases. For your example table, phi and Cramer's v = 0.4, which squared is 0.16. Cohen (1988) also argues that relatively small effect sizes can be meaningful in some cases. I think the obvious case here is when you're talking about being alive or dead, but also I think in social sciences in general, a relatively small effect can be meaningful considering how complex people are. $\endgroup$ – Sal Mangiafico Dec 7 '17 at 18:02
3
$\begingroup$

This is calculated directly from the formula for binomial variance, $np(1-p)$. First, for the null model, not using treatment as predictor, the estimated $p$ is $50/100=0.5$. The variance of the observation "total number alive" is then $np(1-p)=100\cdot 0.5 \cdot 0.5=25$. Under the alternative model, using the treatment predictor, we have two probability parameters, for each of the two groups defined by the treatment, which is $15/50=0.3$ and $35/50=0.7$, leading to a total binomial variance $50 \cdot 0.3 \cdot 0.7 + 50 \cdot 0.7 \cdot 0.3 = 21$. So the reduction in variance by the model is $25-21=4$, which is indeed $16\%$ of the total variance: $4/25=0.16$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.