I am building a logistic multiple regression with 5 potential variables candidates. I read this post and fit all 2^5 = 32 possible combinations of explanatory variables and chose the best model by AIC. However the ‘best’ model does not include any significant variable whereas some other models (with higher AIC) do have some significant variables. I am not statistician and do not understand why such situation. Thank you for any comments, clarification or guidance for another approach if needs be.
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Objective is to look at for an association not for a prediction $\endgroup$– user38000Jan 27, 2014 at 10:04
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
15
This is just a personal opinion, so take it with a grain of salt.
There are a few ways to approach model selection. One of the ways is to construct all possible models and select the one that is 'the best' according to some criterion, e.g. AIC. This is model dredging and is frowned upon by some people because it doesn't incorporate any existing knowledge and maximizes the chance to find a significant model by chance alone. This is basically what you're doing. That being said, data dredging is probably fine if you are exploring an unknown area of research where there is very little or no theoretical knowledge.
Another way would be to, based on what you know of the phenomenon you're trying to model, construct a few valid hypotheses and then construct a model to test these hypotheses. When you compare these models (for example using AIC), you can say something along the lines of "based on what is already known, out of these hypotheses, this hypothesis (or hypotheses) appears describe my data best". You can also take model parameters of a few best models, average them (weighted average) et voila, you have an "average" model that is a compromise between a few good models. This is called the information-theoretic approach. In biology, a book by Burnham and Anderson (Model selection and multimodel inference) is an advocate of this approach. My memory isn't the best, but I think Gentle introduction to MARK has a chapter on this (with a technical part for this specific program on how to model average).
answered Jan 27, 2014 at 10:08
-
$\begingroup$ Thanks @Roman for this comprehensive review. If I would go for the expertise knowledge as stated in your second point, how should I consider or interpret a best model with lower AIC but with no significant variables whereas another model with higher AIC include significant variables? $\endgroup$ Jan 27, 2014 at 10:44
-
$\begingroup$ "it doesn't incorporate any existing knowledge and maximizes the chance to find a significant model by chance alone. This is basically what you're doing." It reads as something bad. Computational costs permitting, isn't the approach used by the OP the best one for model selection? $\endgroup$– user603Jan 27, 2014 at 10:50
-
$\begingroup$ @user603: thanks. Am I wrong if I then consider the model with at least one significant variable with the lower AIC among the subset of models first selected by expert knowledge (rather than the model with the lower AIC even no significant variables). $\endgroup$ Jan 27, 2014 at 11:30
-
3$\begingroup$ This kind of dredging is NOT OK. There are no principles in statistics that back up the approach you have outlined. A few simulations would expose the damage that variable selection does. In general, variable selection without penalization is invalid. $\endgroup$ Jan 27, 2014 at 13:22
-
2$\begingroup$ In my experience, if you are just interested in predictive performance, don't perform any feature selection, but use penalised regression (c.f. ridge regression), see my answer to the question referenced in the question above. It is all too easy to over-fit the feature selection criterion and end up with a model that performs worse than the one with all of the features. You might find this paper useful jstor.org/stable/2347628 $\endgroup$ Jan 27, 2014 at 15:14