# Maximum Likelihood Estimation

If $V_1, V_2,\ldots V_{n_1}$ and $W_1, W_2,\ldots,W_{n_2}$ are independent random samples of size $n_1$ and $n_2$ from normal populations with the means $\mu_1$, $\mu_2$ and the common variance $\sigma^2$, find maximum likelihood estimators for $\mu_1, \mu_2$ and $\sigma^2$.

My idea is to separately find FOC for sample 1 and for sample 2 and then use the 4 equations I'll found to estimate the parameters. Any suggestions? Furthermore: is it a problem the difference in sample size?

Thanks!

• Well, note that $\mu_1$ is only present in the joint pdf of $V_1,\ldots, V_{n_1}$. So it will be the usual mle. Likewise $\mu_2$. Plug in the mle-s of $\mu$'s and then find the mle of $\sigma^2$. – Sayan Nov 16 '14 at 13:50
• Please add the self-study tag and read its tag wiki – Glen_b Nov 17 '14 at 22:27

According to the question, it is a an assumed fact that both populations have common variance, and not something one wishes to test. Maximum likelihood estimators can be derived as usual either from the two samples separately, or by pooling them, in which case we will have an independent but non-identically distributed sample and corresponding log-likelihood, something that nevertheless creates no special issues. So, more than deriving the MLEs (which is straightforward), I would say that this is a good example in order to examine whether pooling samples ("unite and conquer"?) is more beneficial than keeping the samples separate ("divide and conquer"?). But "more beneficial" according to which criteria?

We will discuss them as we go along.

Note that we need both sample sizes to be larger than unity, $n_1 >1, n_2 > 1$, otherwise the variance estimator will equal zero.

If we keep the samples separate we will obtain

$$\hat \mu_v = \frac 1{n_1}\sum_{i=1}^{n_1}v_i,\;\;\; \hat \sigma^2_1 = \frac 1{n_1}\sum_{i=1}^{n_1}(v_i-\hat \mu_v)^2$$

and $$\hat \mu_w = \frac 1{n_2}\sum_{i=1}^{n_2}w_i,\;\;\; \hat \sigma^2_2 = \frac 1{n_2}\sum_{i=1}^{n_2}(w_i-\hat \mu_w)^2$$

The MLEs for the means will be unbiased, efficient, consistent and asymptotically normal.

The variance estimators will be biased, consistent and asymptotically normal (see this post, which holds in general, even for normal samples).

Since we have bias here, it is an easy thought to turn to Mean Squared Error. The populations are normal, so we also have a finite-sample result:

$$\frac {n_i\hat \sigma^2_i}{\sigma^2} \sim \chi^2_{n_i-1} \Rightarrow \hat \sigma^2_i \sim \operatorname{Gamma}(k_i,\theta_i),\;\; k_i = \frac {n_i-1}{2},\;\; \theta_i = \frac {2\sigma^2}{n_i},\;\;i=1,2$$

Therefore we can calculate the Mean Squared Error (MSE) as

$$MSE(\hat \sigma^2_i) = \text{Var}(\hat \sigma^2_i)+\left[B(\hat \sigma^2_i)\right]^2 = \frac{2(n_i-1)}{n_i^2} \sigma^4 + \frac 1{n_i^2}\sigma^4 = \frac{2n_i-1}{n_i^2} \sigma^4$$

We turn now to the pooled-samples case.
It is easy to verify that the MLE's for the two means will be identical with the separate-samples approach. So as regards these estimators, pooling the two samples or not, makes no difference as regards the functional form of the estimators, or their properties.

But the variance estimator will be different. It is also rather easy to derive that

$$\hat \sigma^2_p = \frac{n_1}{n_1+n_2}\hat \sigma^2_1+\frac{n_2}{n_1+n_2}\hat \sigma^2_2$$

This is also a biased an consistent estimator, and also asymptotically normal, being the convex combination of two asymptotically normal variables.

Turning to the issue of bias and Mean Squared Error, since the two separate-samples estimators are independent we have that

$$\text{Var}(\hat \sigma^2_p) = \frac{n_1^2}{(n_1+n_2)^2}\frac{2(n_1-1)}{n_1^2} \sigma^4+\frac{n_2^2}{(n_1+n_2)^2}\frac{2(n_2-1)}{n_2^2}\sigma^4 = \frac {2n_1+2n_2-4}{(n_1+n_2)^2}\sigma^4$$

and

$$B\left(\hat \sigma^2_p\right) = \frac{n_1}{n_1+n_2}E(\hat \sigma^2_1)+\frac{n_2}{n_1+n_2}E(\hat \sigma^2_2) - \sigma^2 = \frac {-2}{n_1+n_2} \sigma^2$$

So the MSE here is

$$MSE(\hat \sigma^2_p) = \frac {2n_1+2n_2-4}{(n_1+n_2)^2}\sigma^4+\frac {4}{(n_1+n_2)^2} \sigma^4 = \frac {2}{n_1+n_2}\sigma^4$$

In order for sample-pooling to be superior in MSE terms we want that

$$MSE(\hat \sigma^2_p) < MSE(\hat \sigma^2_i), i=1,2$$

$$\Rightarrow \frac {2}{n_1+n_2}\sigma^4 < \frac{2n_i-1}{n_i^2} \sigma^4 \Rightarrow 2n_i^2 < 2n_in_1 - n_1 + 2n_in_2 - n_2$$

This reduces to the same condition for either $i=1$ or $i=2$, namely $$0 < - n_1 + 2n_1n_2 - n_2 \Rightarrow \frac {n_1+n_2}{n_1n_2} < 2 \Rightarrow \frac 1{n_2} + \frac {1}{n_1} < 2$$

which holds, since both sample sizes are strictly higher than unity.

Therefore we conclude, that "unite & conquer" is the MSE-efficient approach here.

But we will lose something: if $n_1 \neq n_2$ the pooled-sample variance estimator does not give a Gamma finite sample distributional result, because it is the linear combination of two Gamma random variables with different scale parameters (different $\theta_i$'s). This does not result into a Gamma, but into a rather complicated infinite sum expression (see this paper). Which means that for conducting tests related to the pooled-sample variance estimator, we will have to resort to the asymptotic normality result.

Alternatively, if the difference between $n_1$ and $n_2$ is not large, and both samples have respectable sizes, we may even consider dropping observations from the larger sample in order to make $n_1 =n_2$ and preserve the Gamma distribution result.

• Wouldn't you be able to avoid such a decision with a Bayesian model? Obviously with the trade-off that it requires more "assumptions" – shadowtalker Nov 16 '14 at 15:15
• @ssdecontrol That would be an interesting answer to post! – Alecos Papadopoulos Nov 16 '14 at 15:33
• It would, although I'm not comfortable enough with Bayes to write it myself. Maybe this evening would be a good time to break open my copy of Gelman & Hill – shadowtalker Nov 16 '14 at 15:36
• Doesn't the question ask for maximum likelihood estimators? Where in this answer do you employ the maximum likelihood theory? – whuber Nov 17 '14 at 0:25
• @whuber The estimators given are maximum likelihood estimators. One can check it, if one wishes so -not all steps of mathematical derivations are written in all posts. – Alecos Papadopoulos Nov 17 '14 at 0:30

The strength of maximum likelihood estimation is that it is a purely mechanical process. Just set it up and turn the crank.

To avoid ladders of subscripts, let $m=n_1$ and $n=n_2$ and likewise let $\mu=\mu_1$ and $\nu=\mu_2$. The data are given by the $m+n$-tuple

$$(v_1, v_2, \ldots, v_m, w_1, w_2, \ldots, w_n).$$

The assumption that all $m+n$ values are independent implies the likelihood is the product of the probability densities evaluated at each of the $m+n$ values:

\eqalign{ &L(\mu,\nu,\sigma; \mathrm{v}, \mathrm{w}) \\&= \prod_{i=1}^m \left(2\pi\sigma^2\right)^{-1/2}\exp\left(-\frac{(v_i-\mu)^2}{2\sigma^2}\right) \prod_{j=1}^n \left(2\pi\sigma^2\right)^{-1/2}\exp\left(-\frac{(w_j-\nu)^2}{2\sigma^2}\right). }

Taking logarithms gives

\eqalign{ &\Lambda(\mu_1,\mu_2,\sigma; \mathrm{v}, \mathrm{w}) = \log\left(L(\mu_1,\mu_2,\sigma; \mathrm{v}, \mathrm{w})\right)\\ &=-\frac{n+m}{2}\log(2\pi\sigma^2) - \sum_{i=1}^m \frac{(v_i-\mu)^2}{2\sigma^2} - \sum_{j=1}^n \frac{(w_j-\nu)^2}{2\sigma^2}. }

For $\sigma^2 = 0$ this is undefined. Otherwise it is differentiable with partial derivatives

\eqalign{ -\frac{\partial\Lambda}{\partial \mu} &= \frac{1}{\sigma^2}\sum_{i=1}^m \left(v_i-\mu\right) \\ -\frac{\partial\Lambda}{\partial \nu} &= \frac{1}{\sigma^2}\sum_{j=1}^n \left(w_j-\nu\right) \\ -\frac{\partial\Lambda}{\partial \sigma} &=\frac{n+m}{\sigma} - \frac{1}{\sigma^3}\left(\sum_{i=1}^m(v_i-\mu)^2 + \sum_{j=1}^n(w_i-\nu)^2\right). }

The critical points are obtained as the simultaneous zeros of these equations, easily found in sequence as

\eqalign{ \hat\mu &= \frac{1}{m}\sum_{i=1}^m v_i \\ \hat\nu &= \frac{1}{n}\sum_{j=1}^n w_i \\ \hat\sigma^2 &= \frac{1}{n+m}\left(\sum_{i=1}^m (v_i-\hat\mu)^2 + \sum_{j=1}^n (w_j-\hat\nu)^2\right). }

It is straightforward to check this is the unique maximum provided $\hat\sigma^2 \ne 0$.

• I still don't understand our previous discussion in the comments, since the MLEs in your answer are exactly the same as in mine. – Alecos Papadopoulos Nov 17 '14 at 23:17
• @Alecos Of course they're the same! The point is that there is just one sample and there is just one likelihood to describe it, not two. Your discussion of "pooling," MSE, etc., is superfluous and not relevant to obtaining the MLE. As such that is a distraction, obscuring the simplicity of the solution and leading to unnecessary and even unproductive complications (such as your discussion of dropping values from the sample). – whuber Nov 17 '14 at 23:24
• But why you consider as self-understood, that "we have one sample"? The OP herself starts by describing her data as two distinct samples.This is why I went to all the discussion about pooling or not. What if the one sample was collected six months before, and the other one today? We would have run ML estimation based on the one sample, and when the second became available, we would face the issue "run MLE in the new sample alone, or pool the data"? I don't understand why you consider this as a non-issue. – Alecos Papadopoulos Nov 17 '14 at 23:46
• @Alecos It doesn't matter. The statement that the model supposes two separate means and one common SD is dispositive. It does not leave any choices. If you want to challenge the model, that's fine--but it doesn't answer the question. – whuber Nov 17 '14 at 23:58