3
$\begingroup$

I am having a hard time figuring what Nakagawa's R² really "means". I understand that in simple linear regressions, R² indicates the amount of variance in the dependent variable explained by the predictor variables. For the diverse pseudo-R² measures available for different types of non-linear regression models, it is stated in various ressources that they cannot be interpreted in the same manner as the R² for linear regressions. But what do they mean, then?

From experience I know for example, that McFadden's pseudo-R² generally yields lower values than Nagelkerke's pseudo-R². I understand that Nakagawa's pseudo-R² is available for mixed models in two versions: a) conditional, taking into account fixed and random effects and b) marginal, considering only fixed effects. But I have no idea what these numbers do excatly tell me and, more importantly for me, how to deal with them in practice.

Questions:

  • Are there (+- accurate) thresholds for stating that conditional or marginal pseudo-R² is low / high? In other words: is there a sensible starting point to state that a model is somehow "meaningful" (or not)?
  • Are Nakagawa's pseudo-R² values directly comparable between models with different structure (such as sample size, number/porperty of fixed and/or random effects etc.)?
  • Am I right to assume that the combination of "high" conditional and relatively "low" marginal pseudo-R² indicates that the random effects are decisive factors, whereas the fixed effects do not have much explanatory power? (and vice versa: small differences in conditional and marginal pseudo-R² should then indicate low explanatory power of the random effects)?
$\endgroup$
1
  • $\begingroup$ 1. It is only a metric relative to other models. Yes, super-low (<0.01) or super-high (0.99+) susggest something fishy but on their own they don't mean match 2.I would consider no metrics as being comparable when we have different sample sizes. $\endgroup$ – usεr11852 Oct 9 '20 at 12:58
6
$\begingroup$

I'm just leaving this quote from Douglas Bates's reply in the R-Sig-ME mailing list, on 17 Dec 2014 on the question of how to calculate an $R^2$ statistic for generalized linear mixed models, which I believe is required reading for anyone interested in such a thing. Bates is the original author of the lme4package for R and co-author of nlme, as well as co-author of a well-known book on mixed models

I must admit to getting a little twitchy when people speak of the "R2 for GLMMs". R2 for a linear model is well-defined and has many desirable properties. For other models one can define different quantities that reflect some but not all of these properties. But this is not calculating an R2 in the sense of obtaining a number having all the properties that the R2 for linear models does. Usually there are several different ways that such a quantity could be defined. Especially for GLMs and GLMMs before you can define "proportion of response variance explained" you first need to define what you mean by "response variance". The whole point of GLMs and GLMMs is that a simple sum of squares of deviations does not meaningfully reflect the variability in the response because the variance of an individual response depends on its mean.

Confusion about what constitutes R2 or degrees of freedom of any of the other quantities associated with linear models as applied to other models comes from confusing the formula with the concept. Although formulas are derived from models the derivation often involves quite sophisticated mathematics. To avoid a potentially confusing derivation and just "cut to the chase" it is easier to present the formulas. But the formula is not the concept. Generalizing a formula is not equivalent to generalizing the concept. And those formulas are almost never used in practice, especially for generalized linear models, analysis of variance and random effects. I have a "meta-theorem" that the only quantity actually calculated according to the formulas given in introductory texts is the sample mean.

It may seem that I am being a grumpy old man about this, and perhaps I am, but the danger is that people expect an "R2-like" quantity to have all the properties of an R2 for linear models. It can't. There is no way to generalize all the properties to a much more complicated model like a GLMM.

I was once on the committee reviewing a thesis proposal for Ph.D. candidacy. The proposal was to examine I think 9 different formulas that could be considered ways of computing an R2 for a nonlinear regression model to decide which one was "best". Of course, this would be done through a simulation study with only a couple of different models and only a few different sets of parameter values for each. My suggestion that this was an entirely meaningless exercise was not greeted warmly.

$\endgroup$
4
  • 1
    $\begingroup$ I have already read this statement. However, it does not really help me: it tells (more or less unprecisely), what pseudo-R² is not. So, what do I learn then? Not to use pseudo-R² at all? Considering that in many fields of science it is pretty standard to report these values, I'd prefer to actually use pseudo-R² values with a better understanding what they tell me and how I can interpret them. If I put it the other way round: why did people bother to come up with all these complicated calculations and why does everyone report these values if they are more or less nonsense? $\endgroup$ – yenats Oct 12 '20 at 10:46
  • 3
    $\begingroup$ My advice as a practicing statistician with broad experience in mixed models is to not seek an R^2 measure for GLMMs. It is unfortunate that many scientists wish for it, but that doesn't mean it is a good idea. I can go to my doctor and demand that they prescribe me a certain medication; a good doctor would not acquiesce to that. I would treat the demand for an R^2 for a GLMM in a similar way. $\endgroup$ – Robert Long Oct 12 '20 at 11:05
  • $\begingroup$ Ok, thank you very much for this advice, which I appreciate. It seems difficult to elude the demand for an R^2, though. I wonder whether other measures such as the AUC are more reliable and can be presented as an "alternative". The AUC does not pretend to be an indication of the "variance explained by the model", if I got that right. (But I am not sure about what it indicates, instead). $\endgroup$ – yenats Oct 15 '20 at 7:17
  • $\begingroup$ You're welcome. It is a sad fact of life in applied statistics that we regularly clash with scientists, clinicians and other researchers in this kind of way. I'm not sure what you mean by AUC. If you means the area under the RO curve then this is indeed one metric you can use for predictive models. You can also investigate predictive performance on out-of sample data, or by cross validation. $\endgroup$ – Robert Long Oct 15 '20 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.