The only model with a lower AIC than my null model has no control variables

Question

When you choose a final model to predict sth with a logistic regression. Can you just choose a model without any control variables, if that is the only way to make your AIC smaller than that of the null model?

I always read that you should keep controls, but what if some of the controls are there, mostly because I don't know if they have an impact. In the concept, they should somehow, yes. But the sample is so small, that it is no wonder it is not showing.

There are only two independent variables that make the AIC smaller, both behave very much the same no matter which combination of variables I try. If I add a control or any other variable, the AIC gets bigger than the null model.

When you want to predict sth. in that way, how do you choose the final model?

What I can't comprehend is, that there doesn't seem to be any clear rule like: When a, then do b. When c, then do d. At least none that is applicable to the situation and can't be questioned by the person who evaluates it. It's like a dead end.

Answer 1

What is your research question? That should ultimately dictate how you go about your statistical modeling.

As an example, let's say that your outcome variable Y is whether or not a patient experienced at least one migraine during the past month, your independent variable X is whether or not the patient was on prescribed migraine medication and your control variable Z is gender.

Using these variables, you can formulate different models:

Model 0: logit(p) = beta0 

Model 1: logit(p) = beta0 + beta1*X 

Model 2: logit(p) = beta0 + beta1*X + beta2*Z

Model 3: logit(p) = beta0 + beta1*X + beta2*Z + beta3*X*Z

Model 0 postulates that the probability of experiencing a headache in the past month (denoted by p) is the same for all patients, regardless of their gender and whether or not they are on prescription medication for migraines.

Model 1 postulates that the probability of experiencing a headache in the past month depends on whether or not patients were on prescription medication for migraines (but it doesn't depend on gender).

Model 2 postulates that, among patients of the same gender (be it males or females), the probability of experiencing a headache in the past month depends on whether or not patients were on prescription medication for migraines.

Model 3 postulates that the probability of experiencing a headache in the past month is differnt for males and females and that, for each of these genders, it may depend on whether or not patients were on prescription medication for migraines.

If your research question is "What is the effect of taking prescription migraine medication on the probability of experiencing a headache in the past month among patients of the same gender?", you would want to consider Model 2 or Model 3, whichever is supported by the data.

Generally, control variables are kept in the model even when they are not statistically significant, because they allow you to zoom in on the patient sub-population of interest (e.g., patients of the same gender) in order to answer the question of interest (e.g., what is the effect of taking prescription medications for migraines on the probability of having a migraine in the past month among patients of the same gender?).

Even if you are in a predictive setting, the fact that you have control variables indicates that you should keep those in the model while trying to figure out what combination of independent variables (i.e., variables that are of primary interest to the research problem at hand) maximize the predictive power of the model. Then you would be addressing a research question along these lines: "What combination of independent variables best helps predict whether or not a patient will have experienced migraine in the past month, after accounting for their control variables?".

In my view, the only way to avoid making this a dead end is to be clear about the research question you are trying to address.