10
$\begingroup$

I'm not that familiar with this literature, so please forgive me if this is an obvious question.

Since AIC and BIC depend on maximizing the likelihood, it seems that they can only be used to make relative comparisons between a set of models attempting to fit a given data-set. According to my understanding, it wouldn't make sense to calculate the AIC for Model A on data-set 1, calculate the AIC for Model B on data-set 2, and then compare the two AIC values and judge that (for example) Model A fits data-set 1 better than Model B fits data-set 2. Or perhaps I am mistaken and that is a reasonable thing to do. Please let me know.

My question is this: does there exist a model fit statistic that can be used for absolute instead of just relative comparisons? For linear models, something like $R^2$ would work; it has a defined range and discipline specific ideas about what is a "good" value. I'm looking for something more general and thought that I could start by pinging the experts here. I'm sure that someone has thought of this kind of thing before, but I don't quite know the right terms to make a productive search on Google Scholar.

Any help would be appreciated.

Improve this question
$\endgroup$
7
  • $\begingroup$ If model A fits dataset 1 and model B fits dataset 2, there's nothing at all to compare: the models and the data are totally different. So what exactly are you trying to accomplish? BTW, $R^2$ is worse than useless in this regard; for some criticism, see stats.stackexchange.com/questions/13314/… $\endgroup$
    – whuber ♦
    Commented Jul 22, 2011 at 13:47
  • $\begingroup$ What do you mean by something 'more general' could you give an example to the otye types of models you might want to expand to? Some models will be easy to adapt to an $R^2$ approach, e.g. lowess fits, but others will be quite hard, e.g. fits of binomial data. $\endgroup$
    – russellpierce
    Commented Jul 22, 2011 at 14:00
  • $\begingroup$ @whuber Wow, that's an awesome response to the $R^2$ question! But, its inadequacies aside, $R^2$ is used to say that their model is "good" in an "absolute" sense (e.g. "My $R^2$ is such-and-such which is better than what one normally sees..."). I'm looking for a more justified (and general) statistic than $R^2$ to accomplish the same purpose (e.g. "My MagicStatistic is such-and-such which is better...). My first naive thought was to do something like normalizing a k-fold cross validation score, but it doesn't seem like anyone has done such a thing (so its probably not a good idea). $\endgroup$
    – Nathan VanHoudnos
    Commented Jul 22, 2011 at 14:02
  • 3
    $\begingroup$ @Nathan I don't want to sound like I'm harping on a point or obsessed with it--I'm not--but it occurs to me that people who use $R^2$ to claim their model is good in an absolute sense could often be...mistaken. One lesson of $R^2$ is that a model-fitting statistic is interpretable only in the context of the dataset. When two datasets potentially have nothing in common, what would it really mean to compare two such statistics? So, to get started addressing your question, we need to make assumptions about how the two datasets might be related to each other. Any suggestions? $\endgroup$
    – whuber ♦
    Commented Jul 22, 2011 at 14:08
  • 3
    $\begingroup$ The only thing I could imagine in the realm of what you're talking about would be measure of prediction accuracy. The quality of two models on two different data sets could potentially be compared by which one predicts best, although this is not perfect either. $\endgroup$
    – Macro
    Commented Jul 22, 2011 at 14:19

2 Answers 2

Reset to default
3
$\begingroup$

In line with what Macro suggested I think the term you are looking for is a performance measure. Though it is not a safe way to asses predictive power, it is a very usefull way to compare the fitting quality of various models.

An example measure would be the Mean Average Percentage Error, but more of them can easily be found.

Suppose you use SetA with modelA to describe the number of holes in a road, and you use SetB and modelB to describe the number of people in a country, then of course you cannot say that one model is better than the other, but you can at least see which model provides a more accurate description.

Improve this answer
$\endgroup$
1
$\begingroup$

There are some new-ish papers exploring exactly what you are looking for, I think; Nakagawa and Schielzeth (2013) present an R² statistic for mixed-effects models called "R2 GLMM" to define the amount of unexplained variance in a model.

Conditional R²GLMM is interpreted as variance explained by both fixed and random factors;

Marginal R²GLMM represents the variance explained by fixed factors.

In 2014, Johnson updated the equation to account for random slopes models.

Happily, you can easily calculate both marginal and conditional R²GLMM using the package "MuMIn" in R (Barton, 2015).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.