5
$\begingroup$

Suppose that we have a sample of $n$ observations from a bivariate normal distribution with means equal to zero, variances equal to one and correlation equal to $\rho$, and we would like to compute the maximum likelihood estimator of $\rho$. For the density

$$f(x,y;\rho) = \frac{1}{2 \pi \sqrt{1- \rho^2} } \exp\left\{ -\frac{1}{2 (1-\rho^2)} \left( x^2 - 2\rho xy + y^2 \right) \right\} $$

the estimating (score) equation after some simplication turns out to be

$$\rho \left(1 - \rho^2 \right) + \left( 1 + \rho^2 \right) \frac{ \sum_{i=1}^n x_i y_i}{n} - \rho \frac{\sum_{i=1}^n x_i^2 + \sum_{i=1}^n y_i^2 } {n} = 0$$

which is a cubic polynomial in $\rho$. Since the equation changes sign between $\rho = -1 $ and $\rho = 1$, there is at least one root in that interval but there can be as many as three real ones both inside and outside the interval $[-1, 1]$.

E.L. Lehmann in his book Elements of Large-Sample Theory, however, without further explanation claims that the real solutions corresponding to a maximum are at most two. Here is the exact passage,

enter image description here

This is not obvious to me though, so if anyone sees why that is, I would appreciate the insight. Thank you.

$\endgroup$
  • 1
    $\begingroup$ "At most two" is a little strange, since there will be either one or three. Regardless, I wonder whether he isn't making an asymptotic statement, which would be allowed to assume $\sum x_i^2 = \sum y_i^2 \approx n$, in which case two of the roots will approach $\pm i$. $\endgroup$ – whuber Mar 4 '16 at 20:15
  • $\begingroup$ @whuber Thank you. There can be multiplicities in the roots though, right? This allows the possibility of two real roots. When it comes to the asymptotics, he does mention that the probability of more than one roots goes to zero as $n\to \infty$, which I believe is due to your observation. $\endgroup$ – JohnK Mar 4 '16 at 20:22
  • 1
    $\begingroup$ I don't think he has multiplicities in mind, because in general (for actual data) this equation can have three real roots in the interval $[-1,1]$. $\endgroup$ – whuber Mar 4 '16 at 20:23
  • $\begingroup$ @whuber It's an unfortunate fact that his book, although immensely interesting, is known to contain errors. This might be one of them. Since the exact wording often is crucial, I have included the relevant passage from the book. $\endgroup$ – JohnK Mar 4 '16 at 20:32
  • 1
    $\begingroup$ Thank you for posting the exact quotation, because it completely clears up the issue: this score equation, as you know, is for the derivative. Its roots give critical points, of which there can be at most three, because it's a cubic polynomial in $\rho$. No more than two of those can correspond to maxima of the original log likelihood, because you can't have three local maxima in a row not separated by other critical points. Thus Lehmann is perfectly correct. $\endgroup$ – whuber Mar 4 '16 at 20:37
4
$\begingroup$

The question concerns maximizing the likelihood, which is a function of the sole parameter $\rho \in [-1,1]$, by finding its critical points. Among them are zeros of the derivative of the log likelihood. In this case, the derivative of the log likelihood is a rational function of $\rho$: the numerator is a cubic polynomial and the denominator is a quartic polynomial (equal to $(1-\rho^2)^2$). Since any cubic polynomial may have up to three real roots, how can we be assured there is a unique maximum of the likelihood?

Well, we cannot: as Lehmann points out, there may be two local maxima. But no more than that. The reason is that whenever there are two (anywhere among the real numbers, not just in the range $[-1,1]$) then--because any polynomial is differentiable--there must be a local minimum lying between them. That local minimum would correspond to a third root, and that's all there can be.

This argument can be appreciated by looking at graphs of some of the functions that actually show up. In both figures, the blue dotted curve plots the log likelihood, the red dashed curve plots its derivative, and the solid gold curve is the numerator of the derivative--the cubic polynomial in the question.

Figure 1: three critical points

(This situation can only arise when both $\sum x_i^2$ and $\sum y_i^2$ are appreciably less than $n$, which is increasingly unlikely as $n$ grows large. In this figure, both values are $0.08 n$.)

You can see two local maxima of the log likelood, one near each end. Between them, therefore, there must be a local minimum--and it is apparent at the bottom of the shallow "U" curve near $\rho=0$. Three zeros of the derivative and its numerator are visible where the red and gold curves cross the $\rho$ axis: one of them is very close to $0$, one is negative, and the other positive. (This is a situation where the data are so inconsistent with the model that the maximum likelihood method will not do a good job at estimating $\rho$, which in this case is $0.13$.)

The next figure is more realistic. The value of $\rho$ is the same but this time $\sum x_i^2 = n = \sum y_i^2$. The derivative has exactly one real zero and (therefore) the likelihood has at most one local maximum within the interior of the domain, $(-1,1)$.

Figure 2

Note that $\pm 1$ are always critical points (the derivative becomes unbounded as it approaches these endpoints) and always need to be considered. However, it's not hard to show that they will never be local maxima unless the data are perfectly correlated or perfectly anticorrelated.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.