0
$\begingroup$

My question is very easy and probably banal, but I can't understand this concept and I found nothing on internet.

Consider a logistic/logit model, for example with 3 covariates. We want to test the hypothesis that a model without a variable is preferable. We can do this test with the LRT.

My question is: when there is a better fitting, a better adaptation of the model, the log- likelihood is expected to higher or lower? and why?

For example, if the model with 3 variables is preferable to the one with only 2, if we calculate the log-likelihood of both models ( Reduced model and Complete model ) , which is expected to be higher?

$\endgroup$
1

2 Answers 2

1
$\begingroup$

The maximum over a restricted set is mathematically no larger than the maximum over the full set. You can view the maximized likelihood for model with fewer regressors as the maximum over a restricted set.

Specifically, if you have three regressors, the parameters are $(\beta_0, \beta_1, \beta_2, \beta_3)$, and the maximized likelihood is the maximum over all possible combinations of $(\beta_0, \beta_1, \beta_2, \beta_3)$. The restricted model having only one regressor (say $X_1$) has maximized likelihood over the same set of combinations $(\beta_0, \beta_1, \beta_2, \beta_3)$, but restricted so that $\beta_2 = \beta_3 = 0$. The maximum over the restricted set is no larger than the maximum over the unrestricted set; in most cases it is smaller.

Just because the maximized likelihood is smaller does not necessarily mean the model is worse, though. Since this occurrence is a mathematical fact, the unrestricted model will have (ordinary) higher maximized likelihood even when $\beta_2 = \beta_3 =0$ in reality. The likelihood ratio test specifically addresses this issue, providing a reasonable answer to the question as to whether the difference in maximized likelihoods is explainable by chance alone.

Even if $\beta_2 \neq 0$ or $\beta_3 \neq 0$, the model with only $X_1$ still might be better; penalized likelihood and out-of-sample predictions address this issue.

$\endgroup$
0
$\begingroup$

The log likelihood of a model with more covariates will always be larger than a that of a model with fewer covariates. The reason is simple, if $\alpha \subset \beta$, then

$$\max_{\alpha}L(\alpha) \leq \max_{\beta} L(\beta).$$

However, the question is: how much gain is there in adding the covariates that are in $\beta$ but not in $\alpha$?

The answer is not straightforward, but you can use:

  • Akaike or Bayesian information criteria. These penalize the number of parameters.
  • LASSO, which is implemented in the R package glmnet.
  • Likelihood ratio test.
  • As an informal reference, you can also fit the model using glm in R, and take a look at the p-values of the additional variables.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.