Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I am preparing for an exam, I've been reading through Likelihood Ratio Test, but don't get it.

Example 8.2.3 (page 376 in "Statistical Inference" Casella and Berg)

Let ${{x}_{1}},...{{x}_{n}}$ be a random sample from an exponential population with pdf $$f(x|\theta )=\left\{ \begin{array}{*{35}{l}} {{e}^{-(x-\theta )}} & x\ge \theta \\ 0 & x<\theta \\ \end{array} \right.$$ ${{H}_{0}}_{{}}:\theta \le \theta $ versus ${{H}_{1}}:\theta >{{\theta }_{0}}$

The Likelihood function is, $$L(\theta |x)=\left\{ \begin{array}{*{35}{l}} {{e}^{-\Sigma {{x}_{i}}+n\theta }} & \theta \ge {{x}_{(1)}} \\ 0 & \theta <{{x}_{(1)}} \\ \end{array} \right.$$ The book says: Clearly, $L(\theta |x)$ is an increasing function of $\theta$ on $-\infty < \theta \le {{x}_{(1)}}$. (Does this even matter?). Thus, the denominator of $\lambda(x)$, the unrestricted maximum of $L(\theta|x)$, is

$$L({{x}_{1}}|x)={{e}^{-\Sigma{{x}_{i}}+n{{x}_{(1)}}}}$$ If ${x}_{1}\le{\theta}_{0}$, the numerator of $\lambda(x)$ is also $L({x}_{(1)}|x)$. But since we are maximizing $L(\theta|x)$ over $\theta\le{\theta}_{0}$, the numerator of $\lambda(x)$ is $L({\theta}_{0})$ if ${x}_{1}>{\theta}_{0}$. Therefore, the likelihood ratio test statistic is $$\lambda (x)=\left\{ \begin{array}{*{35}{l}} 1 & {{x}_{(1)}}\ge {{\theta }_{0}} \\ {{e}^{-n\left( {{x}_{(1)}}-{{\theta }_{0}} \right)}} & {{x}_{(1)}}<{{\theta }_{0}} \\ \end{array} \right.$$

Rejection region $$\{x:{{x}_{(1)}}\ge {{\theta }_{0}}-\frac{\log c}{n}\}$$

I read through the whole thing, it was long and tedious to follow, I'm quite lost. Can someone else explain this? Maybe with a better/simple toy example? I want to understand this, not just follow through!

Is this a simple concept, I am over-complicating it?

share|improve this question
    
I think that the $L(\theta \mid x)$ should be positive for $x_(1) \geq \theta$, rather than the other way around. –  Charlie May 2 '12 at 17:54
add comment

2 Answers

Clearly, L(θ|x) is an increasing function of θ on −∞<θ≤x1 . (Does this even matter?).

Yes, this matters. Since L() is increasing wrt θ, we want to choose θ as large as we can in order maximize L(), but we have the constraint that θ≤x1, so the largest we can choose θ is x1.

So x1 is our unrestricted maximum likelihood estimator for θ.

In the numerator, however, we have the additional constraint θ≤θ0, so we have to take that into consideration, too.

Two things can happen.

1) We have x1<θ0 and the constraint doesn't matter, so our unrestricted MLE is the same as our restricted MLE, so the likelihoods are the same, and thus we have a likelihood ratio of 1.

2) We have x1>θ0, so we can't choose x1 without violating θ≤θ0. We've already seen that L() is an increasing function wrt θ, so choosing θ as large as we can will maximize L, and in this case, this corresponds to θ=θ0. When we take the ratio of the two likelihoods, we end up with the second term.

The rejection region will be found by inverting the test statistic.

Hope this helps.

To give a little more concrete and simple example, consider:

I flip a coin 10 times and observe 8 heads. I would like to know if the coin is fair.

This can be characterized as a binomial(10,p) distribution, and we are testing H0: p=0.5 vs p not equal 0.5.

In the numerator, I'll plug in p=0.5 to the binomial distribution, since that's p0.

In the denominator, I'll plug in p=0.8 to the same binomial distribution, since that's the MLE.

I'll reject the null hypothesis when this ratio is very small, because the null hypothesis is very unlikely compared to the alternative hypothesis.

share|improve this answer
    
I somewhat get it now. We want to minimize the negative exponent, so we can maximize the likelihood. The requirement is $\theta \le x$, so we pick the smallest x possible, x1 in our case. Am I anything near to be correct? –  user1061210 Apr 2 '12 at 19:30
    
In some sense, yes, but you should be viewing this as choosing the largest θ rather than the smallest x, as the data (x) are given, and we are maximizing with respect to the parameter (θ) –  Gschneider Apr 2 '12 at 19:45
    
Thanks, that's a great point, I need to correct the way I view things. –  user1061210 Apr 2 '12 at 20:34
    
Just worked through to your example. Your p is the the parameter? p0 is an observed value. I can confused when assigning p or p0, and you have 0.5 and 0.8. Could you explain? –  user1061210 Apr 2 '12 at 23:44
    
P is the parameter, an unobservable probability. P0 is the probability under the null hypothesis, 0.5 (50-50 chance of heads). The unconstrained MLE is 0.8 or 8/10, the usual MLE for the binomial distribution. –  Gschneider Apr 3 '12 at 0:12
add comment

I do not follow your example but I can tell you how I was taught the Likelihood ratio and I'm sure this will be more understandable.

A Likelihood ratio test can be used to compare two nested models (this is important). Where the one model is a special case of the other model. As usual the null is the most restrictive case and expresses the baseline model described below. The test can be used whether adding a few parameters increase the fit (expressed by the log likelihood or -2 log likelihood criterium) of the model. For example, you have a model with 2 X'es and you want to know whether a model which has those 2 X'es and 2 additional X'es performs better, statistically. Adding more parameters in your model always improves the log likelihood but you want to know whether this improvement is large enough to be statistically significant.

The Likelihood ratio test uses the difference between the -2 log likelihoods of the base model (here, the model with 2 X'es) and the extended model (the model with 4 X'es). This test statistic is then compared with the Chi-square distribution with DF the number of X'es added.

The models ofcourse must make use of the log likelihood criteria (logit, probit models for example).I think what L(θ|x) means in your example, is the Likelihood function.

I'm by no means very theoretical, more a practitioner but I hope this helps.

More information can be found on Google ie. http://en.wikipedia.org/wiki/Likelihood-ratio_test

share|improve this answer
    
Maybe I wasn't clear, I edited it. I am working with hypothesis testing, are you talking about regression models? Thanks for your answer. –  user1061210 Apr 1 '12 at 22:31
    
Yes I am talking about regression models. I don't think my comment is appropriate. Maybe the Wikipedia link may shed some more light on your question? –  C. Pieters Apr 1 '12 at 22:50
    
yes, Wikipedia definitely gets me started. I have been looking around for references, but these stuff doesn't seem that popular/prevalent on the web. I was expecting to see some very well-written pdf, but no luck. A little surprising. –  user1061210 Apr 2 '12 at 2:37
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.