0
$\begingroup$

I just learned the likelihood ratio test (LRT) method for model selection and worked out some examples. However, I am still a bit confused with the meaning of it.

Basically, for a family of model parametrised by $\theta$, we consider the restricted family of models parametrised by $\theta_0$ and then we propose the version of restricted parameter space as null hypothesis against the alternative hypothesis which is the general $\theta$ version of models. One then consider the ratio \begin{equation} \text{LR} = -2\log \left[ \frac{\max_{\theta\in\Theta_0} L(\theta)}{\max_{\theta\in\Theta} L(\theta)} \right]. \end{equation} It turns out that this LTR test statistic asymptotically approach the $\chi^2$ distribution and hence one can do hypothesis test to see if one can reject the null hypothesis.

Question: I notice that LRT is useful only in the case of nested parameter spaces, that is, $\theta_0$ must be a restricted version of the general $\theta$. Then, I don't see the point to use LRT for model selection.

If $\theta_0$ is a restricted version of $\theta$, then basically they are "the same model", i.e. parametrised by the same parameter(s) $\theta$. If so, why don't we simply use maximum likelihood estimation to estimate the best parameter $\hat{\theta}$? as usually $\theta_0$ is just some fixed value of $\theta$, then "trying to show $\theta_0$ is not a good choice" seems to not really useful, isn't it? Could someone point out what did I miss? thanks!

$\endgroup$
0
$\begingroup$

It's important to notice that the nested space $\Theta_0$ must have lesser dimensionality than the unconstrained space $\Theta$. The most typical constraint is to put some parameter to 0 (or to 1 in some cases), otherwise some parameter can be fixed to correspond to a linear combination of other parameters.

The larger model (without constraints on $\theta$) has always better or equal likelihood than the nested model, the equivalence having place only if the maximum likelihood point $\hat \theta$ actually respects the constraints (which is almost impossible, because $\Theta_0$ has null volume in $\Theta$).

Hence, likelihood alone is not a valid tool for model selection: larger model will always be selected.

Also, it's worth pointing out that LRT is more of a statistical test than a model selection method, so it is a tool inspired by Popper's theories on epistemology, that starts from a null hypothesis, that for some reason you consider true until proven otherwise, and rejects that hypothesis if data depart strongly enough from it.

So, you can use LRT for model selection, a lot of people do it, but if you are more comfortable with other methods like AIC and BIC, it's a good idea to use those, instead of an hypothesis test created with another use in mind.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ thanks for the answer, I think you have touched some crucial points that I was confused with. Indeed, I thought larger model will be always selected and then there is no difference with MLE. So, if we are seeking for the best model parametrised by $\theta$, then MLE does a nice job, and I don't see what's the point to prove how bad other choice like $\theta_0$ is. However, think of LRT as a hypothesis test makes me feel better. $\endgroup$ – chichi Apr 12 at 1:04
  • $\begingroup$ I think you are also missing the whole purpose of model selection. there are reasons to drop some variables, look here. this doesn't mean it's always necessary or even advisable, it is a very overestimated practice. $\endgroup$ – carlo Apr 12 at 9:45
  • $\begingroup$ Thanks for pointing out the variable selection issue. That indeed makes the whole program meaningful. Up to this point, I think this is indeed the best answer. $\endgroup$ – chichi Apr 12 at 10:05
0
$\begingroup$

You are missunderstanding "restricted". $\theta_0$ restricted means that it has less "free" parameters. For instance, some of the parameters in $\theta$ may have been set directly to zero; or restrictions may have been introduced, so that although $\theta$ and $\theta_0$ are superficially of the same dimension, there are less "free parameters" in $\theta_0$.

This difference in free parameters becomes the degrees of freedom of the $\chi^2$.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.