3
$\begingroup$

This question already has an answer here:

I'm having a go at coding a logistic regression model building algorithm and I'd appreciate some advice. I've read in several places (including here) that minimizing both AIC and BIC could be an effective strategy for model evaluation.

My algorithm employs the following (undoubtedly simplistic) approach:

  1. Fit a model with the intercept only, establish AIC and BIC for null model.

  2. Iterate through the list of features, creating test models that consist of each feature plus the intercept.

  3. Evaluate the test models, selecting that model which minimizes both AIC and BIC scores.

  4. Set AIC and BIC threshold to that of current best model values.

  5. Remove the feature selected in current round.

  6. Repeat, iterating through remaining features, creating new sets of test models for comparison (including previously selected features plus intercept), until no further reduction in AIC and BIC values occur, return the model with current lowest scores for AIC and BIC.

I'm sure this is a hopelessly naive approach, and I'd appreciate some feedback. I suppose my basic question is whether I'm completely misusing these criteria. Thus far, it returns a parsimonious model with very low p-values for all features. Examining the features in depth, it certainly provides some useful insights into the data I'm working with. It's definitely narrowed the field, and given that my data set has 40 features (the final model contained only 6), it's obviously preferable to guessing!

$\endgroup$

marked as duplicate by Stephan Kolassa, kjetil b halvorsen, mdewey, Peter Flom Nov 2 '17 at 12:35

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ And what makes you want to do 'model selection' instead of just 'model specification'? $\endgroup$ – Frank Harrell Nov 2 '17 at 1:28
  • $\begingroup$ Thanks for the link, I'll definitely check it out. I'm working with both categorical and interval data, so I've gone with binary logistic regression. $\endgroup$ – mplane Nov 2 '17 at 1:33
  • $\begingroup$ I went ahead and voted to close as a duplicate. If you think the proposed duplicate does not apply to your question, then please edit your question to draw attention to why the proposal is not a duplicate. Thanks! $\endgroup$ – Stephan Kolassa Nov 2 '17 at 8:44