Timeline for Choosing the best model from among different "best" models
Current License: CC BY-SA 4.0
19 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| S Jun 21, 2022 at 13:54 | history | suggested | Glorfindel | CC BY-SA 4.0 |
broken link fixed, cf. https://math.meta.stackexchange.com/a/34713/228959
|
| Jun 21, 2022 at 8:14 | review | Suggested edits | |||
| S Jun 21, 2022 at 13:54 | |||||
| Oct 27, 2011 at 23:01 | comment | added | Frank Harrell | I believe that is incorrect Joris. You should be able to solve for the transformations that relate AIC, BIC, Cp, and P-values from partial tests. And if you think AIC, BIC, and Cp relieve you of multiplicity problems, think again. | |
| Oct 27, 2011 at 15:07 | history | edited | Joris Meys | CC BY-SA 3.0 |
added 2 characters in body
|
| Oct 27, 2011 at 15:06 | comment | added | Joris Meys | @DmitrijCelov Thx for the catch | |
| Oct 27, 2011 at 14:25 | comment | added | Dmitrij Celov | (+1) for this great answer, just a small correction in the prediction section you probably mean "the outcomes for the tenth piece of dataset" not the tenth model? | |
| Oct 27, 2011 at 11:12 | comment | added | Joris Meys | @FrankHarrell which doesn't mean that I'm advocating pro stepwise, in contrary. But your statement is at least formulated a bit strong. | |
| Oct 27, 2011 at 11:10 | comment | added | Joris Meys | @FrankHarrell a little simulation can prove that the correlation between the p-values (presuming you're talking about the F-test or equivalent) and the AIC is nonexistent (0.01 in my simulation). So no, there's no relation between the P-values and the AIC. Same for BIC and Cp. Another little simulation will also prove that one gets pretty different results in a stepwise procedure depending on the criterium you use. So no: Cp, AIC, BIC are in no way just transformations of P-values. In fact, if looking at the formulas I can in no way point to a mathematical link or transformation. | |
| Oct 26, 2011 at 23:32 | history | edited | rolando2 | CC BY-SA 3.0 |
Merely to say "as few predictors as possible" indicates that zero (or perhaps one, in order to have a model at all) will always be best.
|
| Oct 26, 2011 at 19:51 | comment | added | Frank Harrell | Tom then you want a model with no variables in it. None of these model building methods work as advertised. Cp, AIC, BIC are just transformations of P-values, inheriting all their problems. | |
| Oct 26, 2011 at 17:50 | comment | added | Joris Meys | @tom About which one you choose: depends on what the most important criterion is for you. As said, I don't believe in selection methods, and p-values in a selection are completely meaningless. So the most parsimonious model is the one with the fewest variables, but whether that's the best decision is a whole different matter and cannot be solved without a thorough look at the data and the questions behind. So I won't be able to answer you satisfactory. You'll have to think for yourself. | |
| Oct 26, 2011 at 17:49 | comment | added | Joris Meys | @tom Mind that english is not my mothertongue (so I sound more harsh than I intend at times), and that the bigger part of my attitude consists of writing answers to your question. | |
| Oct 26, 2011 at 17:38 | vote | accept | tom | ||
| Oct 26, 2011 at 17:35 | comment | added | tom | the r and the i look like an n. I don't appreciate the attitude. Also the backwards selection is using the fact that all the variables are significant at $\alpha = 0.05$. The other backwards selection is using the lowest Mallows Cp as a criterion. | |
| Oct 26, 2011 at 17:30 | comment | added | Joris Meys | @tom My name is Joris Meys. Not too difficult to write if it's above your head I presume... If you check the Mallow's Cp, you see it is corrected for the number of variables in the formula. Hence if you use the Mallow's Cp as the backwards criterium, you don't end up at a different model. If the purpose is parsimony, I'd personally check the BIC criterion, as the use of that one in general results in more parsimonious models. | |
| Oct 26, 2011 at 17:01 | comment | added | tom | @Jons Mays: Suppose I choose the model with the lowest Mallows Cp (which has $x$ predictor variables). Then I use backwards selection to get another model (With $x-1$ predictor variables). Would I just choose the one with the fewest variables if the purpose is parsimony? | |
| Oct 26, 2011 at 14:08 | comment | added | Joris Meys | @tom : you're comparing apples with oranges. backward selection is a method, Mallows Cp is a criterion. Mallow's Cp can be used as a criterion for backwards selection. And as you can read, I don't do backward selection. If I need to select variables, I use appropriate methods for that. I didn't mention the LASSO and LAR methods Peter Flom referred to, but they're definitely worth a try too. | |
| Oct 26, 2011 at 14:05 | comment | added | tom | Which model would you select between Mallows Cp and backward selection? Also are models with low SSE and significant coefficients good? | |
| Oct 26, 2011 at 12:10 | history | answered | Joris Meys | CC BY-SA 3.0 |