8
$\begingroup$

I have two nested logistic regression models, A and B. A is nested under B. Let's say B has $K$ more features than A. B has a higher log likelihood than A. However the improved likelihood of B is due to the fact that the $K$ features easily overfit the data. If I apply the likelihood ratio test in my case, it suggests that the more complicated model, B, has a significant improvement. So I think that likelihood ratio test is flawed in such a case.

  • How can we determine whether the added features cause the overfitting problem?
  • Does likelihood ratio test always return the correct answer?
Improve this question
$\endgroup$
9
  • 3
    $\begingroup$ Rather depends on the significance level you choose. Note that selection by the AIC, as suggested by @mdewey & RichardHardy, corresponds to a significance level of 0.15 with the LRT when the models are nested. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Commented May 19, 2016 at 18:15
  • $\begingroup$ @Scortchi Interesting. Where does this 0.15 number come from? $\endgroup$
    – amoeba
    Commented May 19, 2016 at 21:46
  • $\begingroup$ It's 0.16, I think (the upper tail value of a chi-squared distribution with 1 df, in R pchisq(2,df=1,lower.tail=FALSE)), and this number is only correct if we are comparing models that differ by 1 parameter/df. $\endgroup$
    – Ben Bolker
    Commented May 19, 2016 at 23:19
  • 1
    $\begingroup$ @Ben Well, it's 15.73% but I don't think there's all that much issue in describing it as "about 15%" or "about 16%"; the "2" in AIC is an asymptotic figure anyway so it probably doesn't pay to be overprecise about whether to call it 15% or 16% $\endgroup$
    – Glen_b
    Commented May 20, 2016 at 1:41
  • 1
    $\begingroup$ The major part of the premise of this question is simply wrong. It is simply not true that: "likelihood ratio test always suggests that the more complicated model, B, has a significant improvement." (It is true that the likelihood of a more complex model will be higher than an nested less complex model, but the LRT is based on the difference of the log-likelihoods and differences in degrees of freedom.) $\endgroup$
    – DWin
    Commented May 21, 2016 at 17:26

3 Answers 3

Reset to default
8
$\begingroup$

Your reasoning is too pessimistic.

Given the $K$ additional features, the LR test statistic will follow an asymptotic $\chi^2$ distribution with $K$ degrees of freedom if the null is true (and other auxiliary assumptions, e.g., a suitable regression setting, weak dependence assumptions etc.), i.e., if the additional predictors in $B$ are just noise features that lead to "overfitting".

The figure below plots the 0.95%-quantiles of the $\chi^2_K$ distribution as a function of $K$, i.e. the value that the LR statistic needs to exceed to reject the null that $A$ is the "good" model.

enter image description here

As you can see, higher and higher values of the test statistic are needed the larger your set in $B$ that "overfits" the data. So the test suitably makes it more difficult for the (inevitable) better fit (or log-likelihood) of the larger model to be judged "sufficiently" large to reject model $A$.

Of course, for any given application of the test, you might get spurious overfitting that is so "good" that you still falsely reject the null. This "type-I" error is however inherent in any statistical test, and will occur in about 5% of the cases in which the null is true if (like in the figure) we use the 95%-quantiles of the test's null distribution as our critical values.

Improve this answer
$\endgroup$
5
$\begingroup$

How can we determine whether the added features cause the overfitting problem?

It depends on what you intend to use your model for.

  • If you indeed are interested in testing a hypothesis that the $K$ extra features have zero coefficients in your model, then the likelihood ratio (LR) test is the relevant tool to use. It is not flawed in this respect, as shown by @ChristophHanck.
  • If you intend to use your model for prediction, you care whether the extra features improve predictive performance. For that it is not sufficient that the features truly belong in the nesting model; their contribution also needs to be estimated with sufficient accuracy. (If their are estimated with poor accuracy, including them in the model may harm rather than help in prediction.) AIC is the relevant measure in this setting, while the LR test is not particularly well suited for it.

How can we determine whether the added features cause the overfitting problem? Does likelihood ratio test always return the correct answer?

As per @ChristophHanck's answer, there is always a possibility for committing a type I error. But you can control the error rate by setting the significance level sufficiently low, e.g. at 5% or 1%.

Improve this answer
$\endgroup$
3
$\begingroup$

I think you are looking for one of the information criteria such as AIC or BIC which penalise you for adding parameters.

https://en.wikipedia.org/wiki/Akaike_information_criterion has some discussion of both of them.

Note that you should only compare them using the same software as they are only defined up to an additive constant so you cannot compare one computed with R with one computed with Stata.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ Good point, plus one should be careful comparing AIC/BIC values from different packages and even different functions within the same package (e.g. ets and Arima in the "forecast" package in R). $\endgroup$
    – Richard Hardy
    Commented May 19, 2016 at 17:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.