22
$\begingroup$

In stats we're doing linear regressions, the very beginnings of them. In general, we know that the higher the $R^2$ the better, but is there ever a scenario where a high $R^2$ would be a useless model?

$\endgroup$

migrated from stackoverflow.com Dec 16 '15 at 2:08

This question came from our site for professional and enthusiast programmers.

  • 8
    $\begingroup$ The answer at stats.stackexchange.com/questions/13314 might give you some ideas. $\endgroup$ – whuber Dec 16 '15 at 2:27
  • 2
    $\begingroup$ There's one situation discussed here, with an example. For example, if you regressed the results of coin1 on coin2 in the example there, you'd get $R^2$ over 85%, but that apparent relationship is entirely spurious. $\endgroup$ – Glen_b Dec 16 '15 at 11:14
  • 2
    $\begingroup$ $R^2$ is not a model. Therefore you should say "...a high $R^2$ would come from a useless model" or something similar rather than "...a high $R^2$ would be a useless model". $\endgroup$ – Richard Hardy Dec 16 '15 at 21:40
  • $\begingroup$ check this link: What is a good value for R squared $\endgroup$ – Haitao Du Mar 2 '17 at 19:15
  • $\begingroup$ A relevant thread: stats.stackexchange.com/q/414349/121522 $\endgroup$ – mkt Jun 25 at 14:15
43
$\begingroup$

Yes. The criteria for evaluating a statistical model depend on the specific problem at hand and aren't some mechanical function of $R^2$ or statistical significance (though they matter). The relevant question is, "does the model help you understand the data?"

Meaningless regressions with high $R^2$

  1. The simplest way to get high $R^2$ is to do some equivalent of regressing right shoes on left shoes. Tell me the size of your right shoe, and I can predict the size of your left shoe with great accuracy. Huge $R^2$! What a great statistical model! Except it means diddly poo. You can get great $R^2$ by putting the same variable on the left and right hand side of a regression, but this huge $R^2$ regression would almost certainly be useless.

  2. There are other cases where including a variable on the right hand side is conceptually the wrong thing to do (even if it raises $R^2$). Let's say you're trying to estimate if some minority group is discriminated against and less likely to get a job. You shouldn't control for whether the company gave a call back after the job application because being less likely to respond to job applications of minorities may be the channel through which discrimination occurs! Adding the wrong control can render your regression meaningless.

  3. You can always increase $R^2$ by adding more regressors! I can keep adding regressors to the right hand side until I get whatever $R^2$ I like. To predict labor earnings, I could add education controls, age controls, quarter fixed effects, zip code fixed effects, occupation fixed effects, firm fixed effects, family fixed effects, pet fixed effects, hair length etc... at some point the controls cease to make sense but $R^2$ keeps going up. Adding everything as a regressor is known as a "kitchen sink" regression. You can get high $R^2$ but may massively overfit the data: your model perfectly predicts the sample used to estimate the model (has high $R^2$) but the estimated model fails horribly on new data.

  4. Same idea can show up in polynomial curve fitting. Give me random data, and I can probably get great $R^2$ by fitting a 200 degree polynomial. On new data though, the estimated polynomial would fail to work because of overfitting. Again, high $R^2$ for the estimated model but estimated model is useless.

  5. Point (3-4) is why we have adjusted $R^2$, which provides some penalty for adding more regressors, but adjusted $R^2$ can typically still be juiced up by overfitting the data. It also has the wonderfully nonsensical feature that it can go negative.

I could also give examples where low $R^2$ is just fine (e.g. estimating betas in asset pricing models) but this post has already gotten quite long. To summarize, the overall question should be something like, "knowing what I know about the problem and about statistics, does this model help me understand/explain the data?" $R^2$ can be a tool to help answer this question, but it's not so simple as models with higher $R^2$ are always better.

$\endgroup$
  • $\begingroup$ +1 for a lot of good points. I'm trying to figure out what to say about the tone.... $\endgroup$ – rolando2 Dec 16 '15 at 12:22
  • 2
    $\begingroup$ +1. Being a bit pedantic though your "always increase" should have been "never decrease". Assuming that one adds an explanatory variable that is independent of the dependent variable the $R^2$ would remain the same. :D $\endgroup$ – usεr11852 Dec 16 '15 at 15:09
  • 2
    $\begingroup$ being even more pedantic: even if the additional explanatory variable is independent, it will typically still add a little to the $R^2$, because the sample partial correlation will generally be nonnegative even under independence. $\endgroup$ – Christoph Hanck Dec 23 '15 at 8:50
7
$\begingroup$

"Higher is better" is a bad rule of thumb for R-square.

Don Morrison wrote some famous articles a few years back demonstrating that R-squares approaching zero could still both actionable and profitable, depending on the industry. For instance, in direct marketing predicting response to a magazine subscription mailing to 10 million households, R-squares in the low single digits can produce profitable campaigns (on an ROI basis) if the mailing is based on the top 2 or 3 deciles of response likelihood.

Another sociologist (whose name escapes me) segmented R-squares by data type noting that wrt survey research, R-squares in the 10-20% range were the norm, whereas for business data, R-squares in the 40-60% range were to be expected. They went on to remark that R-squares of 80-90% or greater were probably in violation of fundamental regression assumptions. However, this author had no experience with marketing mix, time series data or models containing a full set of "causal" features (e.g., the classic 4 "Ps" of price, promotion, place and product) which can and will produce R-squares approaching 100%.

That said, even sensible, benchmarking rules of thumb such as these aren't terribly helpful when dealing with the technically illiterate whose first question about a predictive model will always be, "What's the R-square?"

$\endgroup$
7
$\begingroup$

The other answers offer great theoretical explanations of the many ways R-squared values can be fixed/faked/misleading/etc.. Here is a hands-on demonstration that has always stuck with me, coded in r:

y <- rnorm(10)
x <- sapply(rep(10,8),rnorm)
summary(lm(y~x))

This can provide R-squared values > 0.90. Add enough regressors and even random values can "predict" random values.

$\endgroup$
  • 1
    $\begingroup$ Interesting: contrast set.seed(1) and set.seed(2). $\endgroup$ – PatrickT Apr 15 '18 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy