D.R. Cox and Nanny Wermuth seem to suggest that the coefficient of determination (R squared) is misleading when you have binary responses, in fact if I am understanding well, they are saying that the maximum value it can be is 0.36.

Is this applicable for OLS models with some key significant independent variables taking a binary value?

Further discussion and insights on the topic would be appreciated.


1 Answer 1


The coefficient of determination can be calculated in various ways which coincide for linear regression. (If that's not true, then something in the comparison is not the coefficient of determination.) Away from that case, it gets messier. Various analogues or alternatives are often (but not always) labelled pseudo-. Watch out also for adjusted relatives penalising for using several predictors.

I have found the paper of Zheng and Agresti (2000) to be helpful in this territory.

Zheng and Agresti (2000) discussed the correlation between the response and the fitted or predicted response as a general measure of predictive power for generalized linear models (GLMs). This measure has the advantages of referring to the original scale of measurement, of being applicable to all types of GLM and of being familiar to many users of statistics. Preferably, it should be used as a comparative measure for different models applied to the same data set, given that restrictions on values of the response may imply limitations on its value (see e.g. Cox and Wermuth, 1992).

For an arbitrary GLM, this correlation is invariant under a location-scale transformation and it is the positive square root of the average proportion of variance explained by the predictors. However, again for an arbitrary GLM, it need not equal the positive square root of other definitions of R-square (e.g. Hardin and Hilbe, 2001); and it need not be monotone increasing in the complexity of the predictors, although in practice that is common. The correlation is necessarily sensitive to outliers.

As the predicted is a function of the observed, the correlation calculated from a sample may be expected to be biased upwards. A jackknifed correlation is recommended as one alternative. Zheng and Agresti provide more discussion of this point, including other estimators and a bootstrap approach to providing confidence intervals for the correlation and to estimating the degree of overfitting.

Cox, D.R. and N. Wermuth. 1992. A comment on the coefficient of determination for binary responses. American Statistician 46: 1-4.

Hardin, J. and J. Hilbe. 2001 (and later editions). Generalized Linear Models and Extensions. College Station, TX: Stata Press.

Zheng, B. and A. Agresti. 2000. Summarizing the predictive power of a generalized linear model. Statistics in Medicine 19: 1771-1781.

Note The use of binary predictors in regression need not limit $R^2$ in linear regression. A simple example is the use of one continuous predictor and one binary predictor. As in principle the data could all lie on two straight lines, a value of 1 is achievable.

  • $\begingroup$ So if I am understanding well, if I have one continuous predictor, and 5 binary predictors, it is wrong to assume that "that the maximum value the R squared can be is 0.36". $\endgroup$ Commented Aug 22, 2013 at 14:22
  • $\begingroup$ Yes. I've not read the Cox and Wermuth paper for some years, but I don't think they said or implied that. $\endgroup$
    – Nick Cox
    Commented Aug 22, 2013 at 14:26
  • $\begingroup$ They state clearly "the interpretation [of the R squared] is misleading in linear regressions with binary responses since low value of R^2, roughly .1 are inevitable even if an important relation is present". Yet you seem to suggest that this can be one. What am I getting wrong here. $\endgroup$ Commented Aug 22, 2013 at 14:31
  • 1
    $\begingroup$ Responses are not predictors. $\endgroup$
    – Nick Cox
    Commented Aug 22, 2013 at 14:39
  • 1
    $\begingroup$ I don't know what "analogy" you have in mind. The point is that there is no such analogy. If you are guessing that the Cox-Wermuth limit applies to linear regression with a continuous response and binary predictors, then that's not so. I've already given a counter-example. $\endgroup$
    – Nick Cox
    Commented Aug 22, 2013 at 14:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.