AIC, pseudo-R2, or log likelihood to compare models?

Question

I am comparing the effect of climate, across three different time brackets, on a variable. I am interested in choosing the model that best predicts the variable to answer across which timescale the climate effect most strongly affects my variable. As such, I run three models:

V~climate1

V~climate2

V~climate3

My instinct is to compare their pseudo-r2 values (not adj r2 or r2 as I'm running phylogenetic gls models in nlme) to assess how well each model predicts my variable of interest. Or is AIC a better alternative? My instinct is no, because each model only has one dependent variable, so there are no extra variables to penalise - in other words, I'm not really sure what AIC would tell me here -- and after all, I'm interested in how well these variables explain the data.

I've also come across log likelihood -- is that better than pseudo-r2 for this question?

Thanks for any hints!

Answer 1

AIC is given as $2k-2log(L)$ where $k$ is the number of parameters and $L$ is the max value of the log likelihood function. Aim is to minimise AIC. In the event that all of your models have an equal number of variables you're essential just comparing the log likelihood since everything else would be constant.

Given that your models essentially emit probabilities (rather than just predictions) you can calculate log likelihoods for each model. If you'd like a scale on which to put log likelihood comparisons you could look into using Bayes Factors (e.g. https://en.wikipedia.org/wiki/Bayes_factor) which have various scales associated to them. You could also use the likelihood ratio test: https://en.wikipedia.org/wiki/Likelihood-ratio_test

I don't think you can use pseudo r2 for model comparison because none of the commonly used calculations offer guarantees of consistency between different models.