4
$\begingroup$

I have a general question on model selection strategies in regression models. In my research, the main goal is rarely prediction but almost always estimation of effects of certain variables.

I have learned about using e.g. AIC or BIC, or comparing models using anova. These methods are quite formal and easily understood.

However, in some situations I might be interested in the effect of a certain variable that does not seem to be associated with my outcome variable, but it doesn't make sense to exclude the variable. A recent example is a study I've been working with in which I want to study the effect on a certain drug on mortality caused by fatal overdose, using an extended cox regression model with periods of drug treatment as a time-dependent variable. It seems obvious that previous non-fatal overdoses should be associated with fatal overdose, but in this case it isn't. Using the methods listed above I should exclude the variable, but I think that it doesn't make sense to exclude that variable. I also think that the lack of association between previous non-fatal overdose and fatal overdose is interesting in itself, though it's not the focus of my study.

This was just a simple example, but I'd like som advice in the general case when we have variables that theoretically should be associated with the outcome variable and we want to include them for theoretical reasons, but the tests indicate that we should exclude them. Is it okay to just include the whole bunch of potentially interesting variables and be done with it? Or should I think carefully about what variables are theoretically important and that should be kept regardless of what e.g. AIC tests say, and exclude only "unimportant" variables that makes no difference in AIC? What difference does this do for the conclusions that we draw from our analyses?

| cite | improve this question | |
$\endgroup$
7
$\begingroup$

You should definitely "think carefully about what variables are theoretically important". If you do all this thinking before you run a model and then you run only that one model, then all the results are correct (provided all the assumptions are met).

If, however, you aren't as sure of your model and need to do some exploration, one way to do it is by using a training and test set of data. Do all your exploration on the training set and then give results of your model from the test set.

No set of automatic procedures can replace judgment.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Thank you. But if we consider a situation in which all variables might have theoretical importance, such as e.g. factors in adolescence that might be associated with alcohol use disorders later in life, then I might have a list of 10 or more variables that might be important (and have some support in the literature), and their interactions might also be important. Then I cannot exclude any "unimportant" variables from start. Is that a case in which using a training and a test set sounds reasonable? $\endgroup$ – JonB Aug 29 '15 at 11:46
  • 2
    $\begingroup$ You will need to have a large enough N to avoid overfitting. If you have 10 variables and want to look at all (10*9)/2 = 45 interactions that's 55 independent variables, which is a lot unless you have a pretty large data set. Exactly what to do will depend on the exact situation $\endgroup$ – Peter Flom♦ Aug 29 '15 at 11:53
  • 1
    $\begingroup$ Ah, yes of course, but thanks for pointing that out. I saw your excellent answer in a similar thread that was linked by Scortchi above. Can you recommend a good book or article that discusses these matters? $\endgroup$ – JonB Aug 29 '15 at 12:01
  • 2
    $\begingroup$ Lately there is a great emphasis on model selection. I think this is frequently misplaced and as Peter said we need to spend our effort on (single) model specification. Note that use of AIC or BIC is at its heart the same as using $P$-values, and results in distortions of statistical inference and bias in regression coefficient estimates. $\endgroup$ – Frank Harrell Aug 29 '15 at 12:30
  • 3
    $\begingroup$ Well, there is always @FrankHarrell 's book Regression Modeling Strategies. $\endgroup$ – Peter Flom♦ Aug 29 '15 at 12:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.