0
$\begingroup$

During analysis of a high dimensional dataset (92 cases, 400+ variables) with the goal of statistical inference, I first used a bootstrapped LASSO (bolasso) to select predictors, and then did an OLS/logistic regression on the most often selected predictors. I proceeded by starting from the null model, and included predictors in decreasing order of selection frequency up to the first non significant predictor.

Now, is it possible to get a better fit (as measured by AIC, for example) by including more (non-significant) predictors? And in that case, should I choose the model with only significant predictors, or the one that has the lowest AIC for best inference?

My understanding of it is that if I include a non-significant predictor in the model, it may yield a better fit only by chance, so it is not justified to include it in the model. Is that correct?

$\endgroup$
  • $\begingroup$ I don't think it is possible that including an insinificant regressors would reduce AIC. I do not have a formal proof, but I read some place on Cross Validated that AIC is somehow equivalent to a cutoff $p$-value of around 0.15, which is above any of the conventional significance levels. I also read some other place on Cross Validated that significance testing should not be used for variable selection. $\endgroup$ – Richard Hardy Nov 22 '15 at 20:32
0
$\begingroup$

Increasing the number of parameters almost always improves the goodness of fit but with increase of parameters comes increase in complexity and the problem of over fitting. AIC (Akaike information criteria) penalizes the model for each increase in the parameter. AIC measure the amount of information lost when we use model to approximate the truth (actual distribution). Although we don't know what the truth is but still AIC will give the measure of information lost when we have to choose between different models (in relative sense). Note that you can't say anything about the absolute loss of information since you don't know what the truth is. So choosing a model with high AIC compared to small one is not advisable but if the AIC value are close enough, weighted average of these model can be used for statistical inference. This wikipedia link briefly explains about AIC and the weights.

$\endgroup$
  • 3
    $\begingroup$ I don't think the OP was asking for an explanation for AIC, but rather how it compares against a very specific alternative model building strategy. $\endgroup$ – Cliff AB Nov 21 '15 at 20:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.