Timeline for Significant difference between AIC in generalized mixed models
Current License: CC BY-SA 3.0
9 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Aug 1, 2013 at 17:21 | answer | added | Peter Flom | timeline score: 4 | |
| Aug 1, 2013 at 15:06 | comment | added | aforsa | I'm talking about the value that gives me SPPS when I fit the model. Maybe I used incorrect words, is the percentage of variance that the model explains, isn't it? anyway, also the model could be significant in a model with higher AIC and no significant in a model with lower AIC. Comparing the same data set. This is why I want to know how to distinguish if two models are statistically different using AIC. | |
| Aug 1, 2013 at 14:48 | comment | added | David Marx | When you say that the selected model's accuracy is lower than models with higher AIC, are you testing against your training data or against a held out test set? It's entirely possible for an AIC selected model to perform more poorly on training data but better on new data. AIC is basically a loss function augmented with a regularization term, so the model that has the lowest RSS will not necessarily have the lowest AIC. That's how AIC is designed: you're selecting for parsimony, not just prediction error. | |
| Aug 1, 2013 at 14:36 | comment | added | aforsa | Thank you for your comments. I'm comparing AICc and I'm just wondering if I select the lower AICc but it isn't significantly different from another with higher AICc, that also has higher accuracy, I am making a mistake (they are not statistically different). So, how can I test that?how can I decide which model is the best. | |
| Aug 1, 2013 at 14:35 | history | edited | gung - Reinstate Monica | CC BY-SA 3.0 |
added tags; light editing & formatting
|
| Aug 1, 2013 at 14:30 | review | First posts | |||
| Aug 1, 2013 at 14:35 | |||||
| Aug 1, 2013 at 14:23 | comment | added | EngrStudent | I like to get a distribution of AIC's, not just 1 or 2. I look at the distribution of them when I replicate the fit on the same data. If I have an outlier it means something about the connection between the data, the fit process, and the analytic form. Now it looks like you are trying to come to terms with your parameters acting to counter-balance your accuracy. If you really don't care then use interpolation and get 100% accurate, but likely terrible generalization. Or trade away accuracy "today" for good generalization "tomorrow". BIC? AICc? Replicate count? | |
| Aug 1, 2013 at 14:20 | history | edited | Nick Cox | CC BY-SA 3.0 |
deleted 16 characters in body; edited title
|
| Aug 1, 2013 at 14:13 | history | asked | aforsa | CC BY-SA 3.0 |