8
$\begingroup$

I'm looking for paper(s) that talk about "why low $R^2$ value is acceptable in social science or education research". Please point me to the right journal if you know one.

$\endgroup$
11
$\begingroup$

A paper by Abelson (1985) titled "A variance explanation paradox: When a little is a lot", published in Psychological Bulletin, addresses (part of) this issue. In particular, Abelson shows that the proportion of variance shared between a dichotomous and a continuous variable can be surprisingly small, even when intuition would dictate a very large $R^2$ (he uses the example of whether a baseball batter would hit a ball or not, as a function of the batter's batting average--yielding a whopping $R^2 < .001$).

Abelson goes on to explain that even such a tiny $R^2$ can be meaningful, as long as the effect under investigation can make itself felt over time.

P.S.: I used this paper a few months ago to respond to a reviewer who was unimpressed with our low $R^2$'s, and it hit the mark--our paper is now in press :)


  • Reference: Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97, 129-133.
$\endgroup$
  • 1
    $\begingroup$ Thanks for the info. I did find the paper interesting. Although it shows a counter example for $R^2$ value I'm trying to find a paper/review that discusses the trend/convention in research involving human behavior/performance. – $\endgroup$ – Amin Sep 10 '14 at 15:48
5
$\begingroup$

An arm-waving argument that nevertheless has much force works backwards. What would perfect prediction imply? For example, it would imply that we can predict students' performance exactly by just knowing their age, sex, race, class, etc. Yet we know that is absurd; it contradicts much else of what we know in social science, not to say everyday life. Moreover, although this is a different issue: many of us would not want to live in such a world.

$\endgroup$
  • $\begingroup$ I am not sure of what you mean. First there are probably not any examples in which you have precisly 1 for a $R^2$ when studying human behaviour, it is just a framework. Second, as any statistical result, it is based on large number results. Last, to take your examples, social sciences (unfortunately ?) often tell us that a few variables (say parents' diploma and implication, ethnicity, hours of work) indeed are very good predictors of a student's achievements. $\endgroup$ – Anthony Martin Dec 10 '14 at 14:15
  • 1
    $\begingroup$ I'd rather doubt "very good" in practice. Unless essentially tautological, $R^2$ in social science seems much more likely to be $\sim 0.1$ than $\sim 1$. $\endgroup$ – Nick Cox Dec 10 '14 at 14:28
  • $\begingroup$ I do not have precise instances in education, but for instance Mincer equations (wages predictions) with only two variables (education and experience) can already yield $R^2$ greater than 0.5 $\endgroup$ – Anthony Martin Dec 10 '14 at 14:41
  • 1
    $\begingroup$ That's consistent with my point. In physical science, $R^2 < 0.9$ is often a sign of incompetence or failure to chose a worthwhile problem. The fact that you regard something like $0.5$ as "very good" stems, I suspect, from your knowing that there are always many unknowns. $\endgroup$ – Nick Cox Dec 10 '14 at 14:45
  • $\begingroup$ I think for sure that human behaviour is more complex and sophisticated than a ponctual mass. Now saying that there are missing variables is not contradictory with the fact if you get them you can potentially have a great predictive power (More refined Mincer equations come close to 0.9 for a $R^2$). Plus I precise that in most of the cases we are more interested in the influence of one parameter hence not in the $R^2$. But there are cases we are : some time-series estimates for instance. You found my statement too theoretical ? $\endgroup$ – Anthony Martin Dec 10 '14 at 14:49
3
$\begingroup$

I find your question a bit vague, it probably depends on what you want to do in social science or education research. But more generally, like every indicator, $R^2$ is good for checking what it is designed to check, bad for the rest.

Precisely, $R^2$ can be defined as $R^2 = \frac{SSE}{SST} = 1 - \frac{SSR}{SST}$, so that it explains how much of the data you can explain by your model, how well data fit a statistical model.

  • The domain where it is the most important is when you want to do prediction : if you want to predict your outcome, it is necessary that your model explains nearly all of what is happening if the data.

  • On the contrary, if you are interested -it is often the case- in the influence of one variable/parameter, you do not care at all about the $R^2$, all you care is that your effects are for instance significant, with the hypothesis needed verified.

I have no precise reference in mind, but any introductory econometrics textbook will have a chapter or section on it (e.g. mostly harmless econometrics or Wooldridge's Introductory Econometrics: A Modern Approach).

$\endgroup$
2
$\begingroup$

Abelson's point could be summarised: What is improbable becomes probable in case of sufficiently many repetitions.

Evolution is build on this principle: It is improbable that a mutation would be an advantage to the mutant. But, in case of sufficiently many mutations, it is likely that a few are advantageous. By means of selection and progeny, the improable afterwards becomes probable in the population.

In both cases, there is a selection mechanism that makes success decisive, and failure not a disaster (for the species at least).

Jesper Juul's book about gaming, "The Art of Failure", adds another dimension to Abelson's considerations. Juul's point is that it is not fascinating to play games where you never loose. Actually, there must be a balance between skills and the frequency of failures/successes, before it becomes attactive to play and improve your performance.

Gaming and training ensure that failure is not a disaster, and then the selection mechanism is effective and low R2 values are no problem, they may even be preferable. Inversely, when failure is a disaster, high R2-values are very important.

More generally, R2 values are important where the event is a game changer. Moreover, gamechanging events often cannot be reduced to a binarity, failure/success: The possible outcomes are multiple and have multiple effects. In that case, the outcome has historical/biographical salience.

In case, events are historical and have never happened before, it is basically impossible to estimate R2, even though some analytical description may reduce randomness because history to some extent may resemble itself. In short, you may experience the combination of small R2 and gamechanging events. ... Well, that is life, sometimes ;-)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.