2
$\begingroup$

I have fit several generalized mixed models for a multinomial distribution data in SPSS.

I want to select the best one for explaining my data. But comparing the AIC and selecting the lower AIC model gives me a model without sense and its accuracy is lower than others with higher AIC.

I want to test if there is any statistical difference between the AIC of the different models, because maybe the values are not statistically different and I could select another model with higher accuracy. Do you know how to do that? E.g.: AICa= 1193 (accuracy 81,3%), AICb= 1273 (88,5%)

I know that with R you can test the differences between models, but I can't develop a generalized mixed model with multinomial function in R. Is that possible in SPSS?

Improve this question
$\endgroup$
4
  • $\begingroup$ I like to get a distribution of AIC's, not just 1 or 2. I look at the distribution of them when I replicate the fit on the same data. If I have an outlier it means something about the connection between the data, the fit process, and the analytic form. Now it looks like you are trying to come to terms with your parameters acting to counter-balance your accuracy. If you really don't care then use interpolation and get 100% accurate, but likely terrible generalization. Or trade away accuracy "today" for good generalization "tomorrow". BIC? AICc? Replicate count? $\endgroup$
    – EngrStudent
    Aug 1, 2013 at 14:23
  • $\begingroup$ Thank you for your comments. I'm comparing AICc and I'm just wondering if I select the lower AICc but it isn't significantly different from another with higher AICc, that also has higher accuracy, I am making a mistake (they are not statistically different). So, how can I test that?how can I decide which model is the best. $\endgroup$
    – aforsa
    Aug 1, 2013 at 14:36
  • $\begingroup$ When you say that the selected model's accuracy is lower than models with higher AIC, are you testing against your training data or against a held out test set? It's entirely possible for an AIC selected model to perform more poorly on training data but better on new data. AIC is basically a loss function augmented with a regularization term, so the model that has the lowest RSS will not necessarily have the lowest AIC. That's how AIC is designed: you're selecting for parsimony, not just prediction error. $\endgroup$
    – David Marx
    Aug 1, 2013 at 14:48
  • $\begingroup$ I'm talking about the value that gives me SPPS when I fit the model. Maybe I used incorrect words, is the percentage of variance that the model explains, isn't it? anyway, also the model could be significant in a model with higher AIC and no significant in a model with lower AIC. Comparing the same data set. This is why I want to know how to distinguish if two models are statistically different using AIC. $\endgroup$
    – aforsa
    Aug 1, 2013 at 15:06

1 Answer 1

Reset to default
4
$\begingroup$

Don't choose a model just because it has a better AIC or a better AICc or a better $R^2$ or any other better property if that model doesn't make any sense.

Model selection is an art. It requires a balance of statistical knowledge and substantive knowledge.

Statistics, more generally, is part of a reasoned argument for or against certain propositions. It ought to be designed to improve knowledge, in whatever field you are in and whether this involves exploration, modeling, or whatever.

Part of the point of learning a lot of statistics is to be able to answer interesting questions and make stronger arguments. It is not to let the computer do your thinking for you.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Peter, how do you respond to the methods of non-parametric statistics? Can it be argued that all non-parametric models don't make any sense? Why or why not? $\endgroup$
    – EngrStudent
    Aug 2, 2013 at 0:58
  • $\begingroup$ Why would non-parametric models make more or less sense than parametric ones, as a general matter? $\endgroup$
    – Peter Flom
    Aug 2, 2013 at 10:22
  • $\begingroup$ When you apply something, like a smoothing spline for EDA, to a reasonably large set of data, then you get many parameters per point. In some sense it is both more informative and less substantive. The spline simplifies the representation and can be useful, but it is a case that is less about comparing propositions and more about trying to make sense of the data. It is a case where standing on the computer you can see farther, but this would not be possible without it. $\endgroup$
    – EngrStudent
    Aug 2, 2013 at 14:36
  • 1
    $\begingroup$ I am certainly not opposed to using computers, or computer intensive methods. I am opposed to letting the computer do your thinking for you. $\endgroup$
    – Peter Flom
    Aug 2, 2013 at 14:39
  • 1
    $\begingroup$ An admirable sentiment. I like to quote Krane, John (2005), “A Physicist on Wall Street,” when he calls such things "voodoo". It isn't science. $\endgroup$
    – EngrStudent
    Aug 2, 2013 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.