1
$\begingroup$

I would like to select the best model for predicting breast cancer risk, specifically, it is the comparisons between weight/BMI/height, as other covariates remain the same in all the models. But I got opposite results for nested model selection by using BIC and LRT. Say that the P-value of LRT is <0.05, but the BIC of the richer model is larger than the less rich model, and the delta BIC is larger than 2, which suggests positive evidence.

Q1: Should I use BIC for nested model comparison? Can BIC penalize the overfitting issue adequately for the richer models?

Q2: Can I use BIC for non-nested model selection and LRT for nested model selection? For example, using BIC to select the best model among weight-only/BMI-only/height-only models; using LRT to select the best model between weight+BMI model Vs BMI-only model to see if weight added additional significant information to the BMI-only model.

I know there are many posts about model selections using BIC/AIC/LRT. But none of them really solve my question.

Improve this question
$\endgroup$
4
  • $\begingroup$ See the answer here: stats.stackexchange.com/questions/198799/…. It concerns AIC rather than BIC but the answer is the same. $\endgroup$
    – Spätzle
    Commented Aug 12, 2021 at 4:15
  • $\begingroup$ I've read that post too, and I raised a question there. I did not get the answer to my question... $\endgroup$
    – Zhoufeng
    Commented Aug 12, 2021 at 4:18
  • $\begingroup$ It still depens on your purpose: If you're just looking to find a better explaining model, then surely you should go by the BIC. $\endgroup$
    – Spätzle
    Commented Aug 12, 2021 at 4:21
  • $\begingroup$ I'm looking for a better predictor, BMI/weight. In all the fitted models, the weight-only model had the smallest BIC. what is confusing to me is that the BIC of the BMI-only model is 3-unit smaller than the BMI+weight model, which means BMI is a better predictor than BMI and weight combined. However, the LRT for BMI+weight Versus BMI is significant, which means weight added additional information to the BMI model. How would you explain these results? $\endgroup$
    – Zhoufeng
    Commented Aug 12, 2021 at 4:45

1 Answer 1

Reset to default
1
$\begingroup$

by definition, BMI and weight are related ($BMI=w/h^2$). Taking them together does not add a significant amount of information to your model, hence the unimpressive BIC difference. In large sample sizes, you would almost always find the additional variable significant - that's a key problem of the traditional significance testing and specifically looking only at the p-value.

Recall that p-value is not "the probability that $H_0$ is wrong" but rather "the probability for rejecting $H_0$ in favor of a powerful alternative $H_1$", and then think of how to compute the power of $H_1$.

Nah, I'm kidding. You don't really need to compute it. Just keep in mind that a p-value does not say anything by itself and that with great sample sizes comes great responsibility for being cautios in interpreting the results.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Thanks for your answer. If the weight model is the best based on BIC. And the BIC of weight+BMI model is smaller than the BMI model. Would that make sense to compare the goodness of fit of the two models based on LRT? The result is that the p value is less than 0.05 which could mean weight added additional information to BMI model, If we assume the importance of LRT.. $\endgroup$
    – Zhoufeng
    Commented Aug 12, 2021 at 13:12
  • 1
    $\begingroup$ That is 100% not the correct definition of a pvalue A pvalue is the probability of observing a statistic >= than the one observed given that the null is true Pvalues are probabilities of data not hypotheses $\endgroup$
    – Stefano Nembrini
    Commented May 9, 2023 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.