1
$\begingroup$

I have fit several generalized mixed models for a multinomial distribution data in SPSS.

I want to select the best one for explaining my data. But comparing the AIC and selecting the lower AIC model gives me a model without sense and its accuracy is lower than others with higher AIC.

I want to test if there is any statistical difference between the AIC of the different models, because maybe the values are not statistically different and I could select another model with higher accuracy. Do you know how to do that? E.g.: AICa= 1193 (accuracy 81,3%), AICb= 1273 (88,5%)

I know that with R you can test the differences between models, but I can't develop a generalized mixed model with multinomial function in R. Is that possible in SPSS?

$\endgroup$
  • $\begingroup$ I like to get a distribution of AIC's, not just 1 or 2. I look at the distribution of them when I replicate the fit on the same data. If I have an outlier it means something about the connection between the data, the fit process, and the analytic form. Now it looks like you are trying to come to terms with your parameters acting to counter-balance your accuracy. If you really don't care then use interpolation and get 100% accurate, but likely terrible generalization. Or trade away accuracy "today" for good generalization "tomorrow". BIC? AICc? Replicate count? $\endgroup$ – EngrStudent - Reinstate Monica Aug 1 '13 at 14:23
  • $\begingroup$ Thank you for your comments. I'm comparing AICc and I'm just wondering if I select the lower AICc but it isn't significantly different from another with higher AICc, that also has higher accuracy, I am making a mistake (they are not statistically different). So, how can I test that?how can I decide which model is the best. $\endgroup$ – aforsa Aug 1 '13 at 14:36
  • $\begingroup$ When you say that the selected model's accuracy is lower than models with higher AIC, are you testing against your training data or against a held out test set? It's entirely possible for an AIC selected model to perform more poorly on training data but better on new data. AIC is basically a loss function augmented with a regularization term, so the model that has the lowest RSS will not necessarily have the lowest AIC. That's how AIC is designed: you're selecting for parsimony, not just prediction error. $\endgroup$ – David Marx Aug 1 '13 at 14:48
  • $\begingroup$ I'm talking about the value that gives me SPPS when I fit the model. Maybe I used incorrect words, is the percentage of variance that the model explains, isn't it? anyway, also the model could be significant in a model with higher AIC and no significant in a model with lower AIC. Comparing the same data set. This is why I want to know how to distinguish if two models are statistically different using AIC. $\endgroup$ – aforsa Aug 1 '13 at 15:06
3
$\begingroup$

Don't choose a model just because it has a better AIC or a better AICc or a better $R^2$ or any other better property if that model doesn't make any sense.

Model selection is an art. It requires a balance of statistical knowledge and substantive knowledge.

Statistics, more generally, is part of a reasoned argument for or against certain propositions. It ought to be designed to improve knowledge, in whatever field you are in and whether this involves exploration, modeling, or whatever.

Part of the point of learning a lot of statistics is to be able to answer interesting questions and make stronger arguments. It is not to let the computer do your thinking for you.

$\endgroup$
  • $\begingroup$ Peter, how do you respond to the methods of non-parametric statistics? Can it be argued that all non-parametric models don't make any sense? Why or why not? $\endgroup$ – EngrStudent - Reinstate Monica Aug 2 '13 at 0:58
  • $\begingroup$ Why would non-parametric models make more or less sense than parametric ones, as a general matter? $\endgroup$ – Peter Flom - Reinstate Monica Aug 2 '13 at 10:22
  • $\begingroup$ When you apply something, like a smoothing spline for EDA, to a reasonably large set of data, then you get many parameters per point. In some sense it is both more informative and less substantive. The spline simplifies the representation and can be useful, but it is a case that is less about comparing propositions and more about trying to make sense of the data. It is a case where standing on the computer you can see farther, but this would not be possible without it. $\endgroup$ – EngrStudent - Reinstate Monica Aug 2 '13 at 14:36
  • 1
    $\begingroup$ I am certainly not opposed to using computers, or computer intensive methods. I am opposed to letting the computer do your thinking for you. $\endgroup$ – Peter Flom - Reinstate Monica Aug 2 '13 at 14:39
  • 1
    $\begingroup$ An admirable sentiment. I like to quote Krane, John (2005), “A Physicist on Wall Street,” when he calls such things "voodoo". It isn't science. $\endgroup$ – EngrStudent - Reinstate Monica Aug 2 '13 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.