# Model selection, issues of judgement

I have a general question on model selection strategies in regression models. In my research, the main goal is rarely prediction but almost always estimation of effects of certain variables.

I have learned about using e.g. AIC or BIC, or comparing models using anova. These methods are quite formal and easily understood.

However, in some situations I might be interested in the effect of a certain variable that does not seem to be associated with my outcome variable, but it doesn't make sense to exclude the variable. A recent example is a study I've been working with in which I want to study the effect on a certain drug on mortality caused by fatal overdose, using an extended cox regression model with periods of drug treatment as a time-dependent variable. It seems obvious that previous non-fatal overdoses should be associated with fatal overdose, but in this case it isn't. Using the methods listed above I should exclude the variable, but I think that it doesn't make sense to exclude that variable. I also think that the lack of association between previous non-fatal overdose and fatal overdose is interesting in itself, though it's not the focus of my study.

This was just a simple example, but I'd like som advice in the general case when we have variables that theoretically should be associated with the outcome variable and we want to include them for theoretical reasons, but the tests indicate that we should exclude them. Is it okay to just include the whole bunch of potentially interesting variables and be done with it? Or should I think carefully about what variables are theoretically important and that should be kept regardless of what e.g. AIC tests say, and exclude only "unimportant" variables that makes no difference in AIC? What difference does this do for the conclusions that we draw from our analyses?

• Lately there is a great emphasis on model selection. I think this is frequently misplaced and as Peter said we need to spend our effort on (single) model specification. Note that use of AIC or BIC is at its heart the same as using $P$-values, and results in distortions of statistical inference and bias in regression coefficient estimates. – Frank Harrell Aug 29 '15 at 12:30