3
$\begingroup$

I have used the R lme function (nlme package) to construct linear mixed models, with a single random effect (as a random intercept) and a varIdent variance structure on a fixed effect (that is a factor).

The data is from a commercial fishery and I am looking to see which are the most important variables that need to be reported for this fishery. I have 3 biological variables (total length, weight, proportion of females) and two operational variables (effort and season) (unfortunately I can't post my data as it's confidential and I haven't managed to reproduce my problem with dummy data).

I have used the AICc to select the 'best' models from a list of a priori models and I am now calculating the R-squared values from each of the 'best' models (using the code from https://github.com/jslefche/rsquared.glmer/blob/master/rsquaredglmm.R from the blog post http://jonlefcheck.net/2013/03/13/r2-for-linear-mixed-effects-models/ and taken from Nakagawa and Schielzeth (2013) r2 for GLMMs). I have 6 models with a AICc < 10 and I've used that as my cut off as it gives me a balanced number of models to the calculate the relative importance of each variable of interest using the sum Akaike's weights (the purpose of the modelling).

However, the R-squared values are lower for my 'best' models (Akaike weights between 0.5-0.8; marginal $R^2$ = 0.48) than the model with all my fixed effects (Akaike weights 0.00; marginal $R2^$ = 0.89).

I've re-run the model selection using BIC and I get the same problem. My question is: Are most 'best' models the models that have a lower $R^2$ value (and lower AICc) or have higher R2 values but higher AICc values?

$\endgroup$
  • $\begingroup$ Though I'm not familiar with the specifics of your approach, I would avoid parsimony without shrinkage. If LASSO is appropriate, then that might be the easiest solution to your problem. $\endgroup$ – user44764 Jul 9 '14 at 13:21
3
$\begingroup$

Is your goal model parsimony or the predictive power of the model? If parsimony, then use AIC, if predictive power then $R^2$. Usually the answer is similar, but if you are comparing models with very similar $R^2$ or a number of low quality predictors the answers can be different. This is why in regular regression we tend to look at adjusted $R^2$ rather than just $R^2$, that is, because the adjusted value penalizes $R^2$ to adjust for the variance one might expect to be explained by chance if a predictor was not really effective at all. As the author says in the blog post "although I should note that [$R^2$ is] a poor tool for model selection, since it almost always favors the most complex models".

P.S. If you are interested in the fixed effects relative to the overall variance (regardless of nesting factor), then you probably want to be looking at the marginal $R^2$.

$\endgroup$
  • $\begingroup$ It is probably not a good idea to recommend using $R^2$ for the goal of maximizing predictive power when (as you point out yourself) it is a poor tool for model selection. And that probably undersells it. $R^2$ is terrible for model selection, as it will increase as predictors are added i.e. overfitting is guaranteed. Adjusted $R^2$ is better, but not by a lot. If the goal is maximizing predictive power, then that is better evaluated using prediction error metrics (MAE, RMSE, etc) using crossvalidation or a holdout dataset. $\endgroup$ – mkt - Reinstate Monica Jul 1 at 15:02
0
$\begingroup$

R^2 tell you how much of the variance a model is explained. AIC is based on the KL distance and compares models relative to one another. For instance if you wanted to compare via R^2 you'd want to know if the change in R^2 is significant, and if not by parsimony take the simpler model, and if so take the more complex model assuming your simpler model is nested in the more complex model.

Parsimony is important when considering the predictive power, since if you're model is too complex it may memorize your current set of data and seem to perform very well, however, as a result not be able to handle differences in a new set of data over estimating the predictive power.

So my repose is don't compare AIC to R^2, compare AIC with change in R^2

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.