# Comparison between Nested and Non-nested Model: BIC and LRT

I would like to select the best model for predicting breast cancer risk, specifically, it is the comparisons between weight/BMI/height, as other covariates remain the same in all the models. But I got opposite results for nested model selection by using BIC and LRT. Say that the P-value of LRT is <0.05, but the BIC of the richer model is larger than the less rich model, and the delta BIC is larger than 2, which suggests positive evidence.

Q1: Should I use BIC for nested model comparison? Can BIC penalize the overfitting issue adequately for the richer models?

Q2: Can I use BIC for non-nested model selection and LRT for nested model selection? For example, using BIC to select the best model among weight-only/BMI-only/height-only models; using LRT to select the best model between weight+BMI model Vs BMI-only model to see if weight added additional significant information to the BMI-only model.

I know there are many posts about model selections using BIC/AIC/LRT. But none of them really solve my question.

• See the answer here: stats.stackexchange.com/questions/198799/…. It concerns AIC rather than BIC but the answer is the same. Aug 12 at 4:15
• I've read that post too, and I raised a question there. I did not get the answer to my question... Aug 12 at 4:18
• It still depens on your purpose: If you're just looking to find a better explaining model, then surely you should go by the BIC. Aug 12 at 4:21
• I'm looking for a better predictor, BMI/weight. In all the fitted models, the weight-only model had the smallest BIC. what is confusing to me is that the BIC of the BMI-only model is 3-unit smaller than the BMI+weight model, which means BMI is a better predictor than BMI and weight combined. However, the LRT for BMI+weight Versus BMI is significant, which means weight added additional information to the BMI model. How would you explain these results? Aug 12 at 4:45

by definition, BMI and weight are related ($$BMI=w/h^2$$). Taking them together does not add a significant amount of information to your model, hence the unimpressive BIC difference. In large sample sizes, you would almost always find the additional variable significant - that's a key problem of the traditional significance testing and specifically looking only at the p-value.
Recall that p-value is not "the probability that $$H_0$$ is wrong" but rather "the probability for rejecting $$H_0$$ in favor of a powerful alternative $$H_1$$", and then think of how to compute the power of $$H_1$$.