1
$\begingroup$

During analysis of a high dimensional dataset (92 cases, 400+ variables) with the goal of statistical inference, I first used a bootstrapped LASSO (bolasso) to select predictors, and then did an OLS/logistic regression on the most often selected predictors. I proceeded by starting from the null model, and included predictors in decreasing order of selection frequency up to the first non-significant predictor.

Now, is it possible to get a better fit (as measured by AIC, for example) by including more (non-significant) predictors? And in that case, should I choose the model with only significant predictors, or the one that has the lowest AIC for best inference?

My understanding of it is that if I include a non-significant predictor in the model, it may yield a better fit only by chance, so it is not justified to include it in the model. Is that correct?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ I don't think it is possible that including an insinificant regressors would reduce AIC. I do not have a formal proof, but I read some place on Cross Validated that AIC is somehow equivalent to a cutoff $p$-value of around 0.15, which is above any of the conventional significance levels. I also read some other place on Cross Validated that significance testing should not be used for variable selection. $\endgroup$
    – Richard Hardy
    Commented Nov 22, 2015 at 20:32
  • 3
    $\begingroup$ It’s been five years, but what is your goal in modeling the data? $\endgroup$
    – Dave
    Commented Jul 4, 2020 at 14:59
  • 1
    $\begingroup$ "Statistical significance" is a bad goal and the sample size is too small by a factor of 100 for reliably determining which predictors are 'important". And the use if AIC or any method to remove variables up front will destroy the meaning of p-values. $\endgroup$
    – Frank Harrell
    Commented Dec 5, 2020 at 12:02
  • $\begingroup$ @FrankHarrell yes. Unfortunately, I am dealing with medical academics, here. They are not too much concerned about whether the results make sense. $\endgroup$
    – Raoul
    Commented Dec 6, 2020 at 14:19
  • 2
    $\begingroup$ You can't let yourself be involved in a project where the investigators are doing bad science, if you can't change them. This project badly needs a heavy dose of unsupervised learning (data reduction masked to Y) to collapse the questions into what the sample size will support. $\endgroup$
    – Frank Harrell
    Commented Dec 6, 2020 at 17:01

1 Answer 1

Reset to default
0
$\begingroup$

Increasing the number of parameters almost always improves the goodness of fit but with increase of parameters comes increase in complexity and the problem of over fitting. AIC (Akaike information criterion) penalizes the model for each increase in the parameter. AIC measures the amount of information lost when we use a model to approximate the truth (actual distribution). Although we don't know what the truth is but still AIC will give the measure of information lost when we have to choose between different models (in relative sense). Note that you can't say anything about the absolute loss of information since you don't know what the truth is. So choosing a model with high AIC compared to small one is not advisable but if the AIC value is close enough, a weighted average of these models can be used for statistical inference. This wikipedia link briefly explains about AIC and the weights.

Improve this answer
$\endgroup$
1
  • 6
    $\begingroup$ I don't think the OP was asking for an explanation for AIC, but rather how it compares against a very specific alternative model building strategy. $\endgroup$
    – Cliff AB
    Commented Nov 21, 2015 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.