8
$\begingroup$

I'm looking for paper(s) that talk about "why low $R^2$ value is acceptable in social science or education research". Please point me to the right journal if you know one.

Improve this question
$\endgroup$
1

4 Answers 4

Reset to default
11
$\begingroup$

A paper by Abelson (1985) titled "A variance explanation paradox: When a little is a lot", published in Psychological Bulletin, addresses (part of) this issue. In particular, Abelson shows that the proportion of variance shared between a dichotomous and a continuous variable can be surprisingly small, even when intuition would dictate a very large $R^2$ (he uses the example of whether a baseball batter would hit a ball or not, as a function of the batter's batting average--yielding a whopping $R^2 < .001$).

Abelson goes on to explain that even such a tiny $R^2$ can be meaningful, as long as the effect under investigation can make itself felt over time.

P.S.: I used this paper a few months ago to respond to a reviewer who was unimpressed with our low $R^2$'s, and it hit the mark--our paper is now in press :)

  • Reference: Abelson, R. P. (1985). A variance explanation paradox: When a little is a lot. Psychological Bulletin, 97, 129-133.
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the info. I did find the paper interesting. Although it shows a counter example for $R^2$ value I'm trying to find a paper/review that discusses the trend/convention in research involving human behavior/performance. – $\endgroup$
    – Amin
    Sep 10, 2014 at 15:48
6
$\begingroup$

An arm-waving argument that nevertheless has much force works backwards. What would perfect prediction imply? For example, it would imply that we can predict students' performance exactly by just knowing their age, sex, race, class, etc. Yet we know that is absurd; it contradicts much else of what we know in social science, not to say everyday life. Moreover, although this is a different issue: many of us would not want to live in such a world.

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ I am not sure of what you mean. First there are probably not any examples in which you have precisly 1 for a $R^2$ when studying human behaviour, it is just a framework. Second, as any statistical result, it is based on large number results. Last, to take your examples, social sciences (unfortunately ?) often tell us that a few variables (say parents' diploma and implication, ethnicity, hours of work) indeed are very good predictors of a student's achievements. $\endgroup$
    – Anthony Martin
    Dec 10, 2014 at 14:15
  • 2
    $\begingroup$ I'd rather doubt "very good" in practice. Unless essentially tautological, $R^2$ in social science seems much more likely to be $\sim 0.1$ than $\sim 1$. $\endgroup$
    – Nick Cox
    Dec 10, 2014 at 14:28
  • 1
    $\begingroup$ I do not have precise instances in education, but for instance Mincer equations (wages predictions) with only two variables (education and experience) can already yield $R^2$ greater than 0.5 $\endgroup$
    – Anthony Martin
    Dec 10, 2014 at 14:41
  • 2
    $\begingroup$ That's consistent with my point. In physical science, $R^2 < 0.9$ is often a sign of incompetence or failure to chose a worthwhile problem. The fact that you regard something like $0.5$ as "very good" stems, I suspect, from your knowing that there are always many unknowns. $\endgroup$
    – Nick Cox
    Dec 10, 2014 at 14:45
  • 1
    $\begingroup$ I don't see any theory in your discussion at all. More importantly, I don't think we're contradicting each other. You're impressed that we can get moderate $R^2$ some times in social science; I am not surprised and I didn't claim otherwise. $\endgroup$
    – Nick Cox
    Dec 10, 2014 at 15:42
4
$\begingroup$

I find your question a bit vague, it probably depends on what you want to do in social science or education research. But more generally, like every indicator, $R^2$ is good for checking what it is designed to check, bad for the rest.

Precisely, $R^2$ can be defined as $R^2 = \frac{SSE}{SST} = 1 - \frac{SSR}{SST}$, so that it explains how much of the data you can explain by your model, how well data fit a statistical model.

  • The domain where it is the most important is when you want to do prediction : if you want to predict your outcome, it is necessary that your model explains nearly all of what is happening if the data.

  • On the contrary, if you are interested -it is often the case- in the influence of one variable/parameter, you do not care at all about the $R^2$, all you care is that your effects are for instance significant, with the hypothesis needed verified.

I have no precise reference in mind, but any introductory econometrics textbook will have a chapter or section on it (e.g. mostly harmless econometrics or Wooldridge's Introductory Econometrics: A Modern Approach).

Improve this answer
$\endgroup$
2
$\begingroup$

Abelson's point could be summarised: What is improbable becomes probable in case of sufficiently many repetitions.

Evolution is build on this principle: It is improbable that a mutation would be an advantage to the mutant. But, in case of sufficiently many mutations, it is likely that a few are advantageous. By means of selection and progeny, the improable afterwards becomes probable in the population.

In both cases, there is a selection mechanism that makes success decisive, and failure not a disaster (for the species at least).

Jesper Juul's book about gaming, "The Art of Failure", adds another dimension to Abelson's considerations. Juul's point is that it is not fascinating to play games where you never loose. Actually, there must be a balance between skills and the frequency of failures/successes, before it becomes attactive to play and improve your performance.

Gaming and training ensure that failure is not a disaster, and then the selection mechanism is effective and low R2 values are no problem, they may even be preferable. Inversely, when failure is a disaster, high R2-values are very important.

More generally, R2 values are important where the event is a game changer. Moreover, gamechanging events often cannot be reduced to a binarity, failure/success: The possible outcomes are multiple and have multiple effects. In that case, the outcome has historical/biographical salience.

In case, events are historical and have never happened before, it is basically impossible to estimate R2, even though some analytical description may reduce randomness because history to some extent may resemble itself. In short, you may experience the combination of small R2 and gamechanging events. ... Well, that is life, sometimes ;-)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.