the meaning of likelihood ratio test

Question

I just learned the likelihood ratio test (LRT) method for model selection and worked out some examples. However, I am still a bit confused with the meaning of it.

Basically, for a family of model parametrised by $\theta$, we consider the restricted family of models parametrised by $\theta_0$ and then we propose the version of restricted parameter space as null hypothesis against the alternative hypothesis which is the general $\theta$ version of models. One then consider the ratio \begin{equation} \text{LR} = -2\log \left[ \frac{\max_{\theta\in\Theta_0} L(\theta)}{\max_{\theta\in\Theta} L(\theta)} \right]. \end{equation} It turns out that this LTR test statistic asymptotically approach the $\chi^2$ distribution and hence one can do hypothesis test to see if one can reject the null hypothesis.

Question: I notice that LRT is useful only in the case of nested parameter spaces, that is, $\theta_0$ must be a restricted version of the general $\theta$. Then, I don't see the point to use LRT for model selection.

If $\theta_0$ is a restricted version of $\theta$, then basically they are "the same model", i.e. parametrised by the same parameter(s) $\theta$. If so, why don't we simply use maximum likelihood estimation to estimate the best parameter $\hat{\theta}$? as usually $\theta_0$ is just some fixed value of $\theta$, then "trying to show $\theta_0$ is not a good choice" seems to not really useful, isn't it? Could someone point out what did I miss? thanks!

Answer 1

It's important to notice that the nested space $\Theta_0$ must have lesser dimensionality than the unconstrained space $\Theta$. The most typical constraint is to put some parameter to 0 (or to 1 in some cases), otherwise some parameter can be fixed to correspond to a linear combination of other parameters.

The larger model (without constraints on $\theta$) has always better or equal likelihood than the nested model, the equivalence having place only if the maximum likelihood point $\hat \theta$ actually respects the constraints (which is almost impossible, because $\Theta_0$ has null volume in $\Theta$).

Hence, likelihood alone is not a valid tool for model selection: larger model will always be selected.

Also, it's worth pointing out that LRT is more of a statistical test than a model selection method, so it is a tool inspired by Popper's theories on epistemology, that starts from a null hypothesis, that for some reason you consider true until proven otherwise, and rejects that hypothesis if data depart strongly enough from it.

So, you can use LRT for model selection, a lot of people do it, but if you are more comfortable with other methods like AIC and BIC, it's a good idea to use those, instead of an hypothesis test created with another use in mind.

Answer 2

You are missunderstanding "restricted". $\theta_0$ restricted means that it has less "free" parameters. For instance, some of the parameters in $\theta$ may have been set directly to zero; or restrictions may have been introduced, so that although $\theta$ and $\theta_0$ are superficially of the same dimension, there are less "free parameters" in $\theta_0$.

This difference in free parameters becomes the degrees of freedom of the $\chi^2$.