Finding MLE of the common $\mu$ from normal samples with two unknown variances

Question

My problem is as follows,

Find the maximum likelihood estimator, $\mu^{MLE}$ from $(m+n)$ samples, where $X_1, \cdots, X_m \sim N(\mu, \sigma_1^2), Y_1, \cdots, Y_n \sim N(\mu, \sigma_2^2),$

where $\sigma_1^2, \sigma_2^2$ : both unknown.

My attempt

$$ \begin{aligned} \log L(\mu) = l(\mu) &= \mathrm{constant}-\frac{1}{2} \left[\sum_{i=1}^m\frac{(x_i - \mu)^2}{\sigma_1^2} + \sum_{j=1}^n\frac{(y_j - \mu)^2}{\sigma_2^2} \right] \\ \frac{\partial l(\mu)}{\partial \mu} &= - \frac{\mu}{\sigma_1^2} \sum_{i=1}^m x_i^2 - \frac{\mu}{\sigma_2^2} \sum_{j=1}^n y_j^2 + \frac{m \bar{x}}{\sigma_1^2} + \frac{n \bar{y}}{\sigma_2^2} \end{aligned} $$

This yields $\hat{\mu^{MLE}} =\frac{ m \bar{x}/\sigma_1^2 +n\bar{y}/\sigma_2^2 }{m /\sigma_1^2 +n/\sigma_2^2}$, but the main problem is, I basically don't know the variances. Any help will be appreciated.

Answer 1

Your attempt at the problem is correct so far, but you have only derived the conditional MLE when the variance parameters are known. To derive the unconditional MLE for the mean parameter, you will need to derive the corresponding equations for the MLEs of the variance parameters and then solve the resulting set of simultaneous equations. This should give you a unique MLE estimating each of the three parameters in the model. Let me show you how to do this.

---

Derivation of the full MLE: For greater clarity, I will denote the variance parameters as $\sigma_x^2$ and $\sigma_y^2$ rather than denoting them with number subscripts. From your specified model, the log-likelihood for your observed data (ignoring an additive constant) can be written as:

$$\begin{equation} \begin{aligned} \ell(\mu,\sigma_x,\sigma_y) &= - m \ln \sigma_x - n \ln \sigma_y -\frac{1}{2} \Bigg[ \sum_{i=1}^m \frac{(x_i - \mu)^2}{\sigma_x^2} + \sum_{i=1}^n \frac{(y_i - \mu)^2}{\sigma_y^2} \Bigg] \\[6pt] &= - m \ln \sigma_x - n \ln \sigma_y -\frac{1}{2} \Bigg[ \frac{1}{\sigma_x^2} \sum_{i=1}^m (x_i^2 - 2 \mu x_i + \mu^2) + \frac{1}{\sigma_y^2} \sum_{i=1}^n (y_i^2 - 2 \mu y_i + \mu^2) \Bigg] \\[6pt] &= - m \ln \sigma_x - n \ln \sigma_y -\frac{1}{2} \Bigg( \frac{1}{\sigma_x^2} \sum_{i=1}^m x_i^2 + \frac{1}{\sigma_y^2} \sum_{i=1}^n y_i^2 \Bigg) \\[6pt] &\quad \quad \quad \quad \quad \quad \quad \text{ } \text{ } + \Bigg( \frac{m\bar{x}}{\sigma_x^2} + \frac{n\bar{y}}{\sigma_y^2} \Bigg) \mu -\frac{1}{2} \Bigg( \frac{m}{\sigma_x^2} + \frac{n}{\sigma_y^2} \Bigg) \mu^2. \\[6pt] \end{aligned} \end{equation}$$

where $\bar{x} = \sum_{i=1}^m x_i / m$ and $\bar{y} = \sum_{i=1}^n y_i / n$ are the sample means of the parts. Hence, your score function consists of the following partial derivatives:

$$\begin{equation} \begin{aligned} \frac{\partial \ell}{\partial \mu}(\mu,\sigma_x,\sigma_y) &= \Bigg( \frac{m\bar{x}}{\sigma_x^2} + \frac{n\bar{y}}{\sigma_y^2} \Bigg) - \Bigg( \frac{m}{\sigma_x^2} + \frac{n}{\sigma_y^2} \Bigg) \mu, \\[10pt] \frac{\partial \ell}{\partial \sigma_x}(\mu,\sigma_x,\sigma_y) &= - \frac{1}{\sigma_x^3} \Bigg( m \sigma_x^2 - \sum_{i=1}^m (x_i - \mu)^2 \Bigg) , \\[10pt] \frac{\partial \ell}{\partial \sigma_y}(\mu,\sigma_x,\sigma_y) &= - \frac{1}{\sigma_y^3} \Bigg( n \sigma_y^2 - \sum_{i=1}^n (y_i - \mu)^2 \Bigg) . \\[10pt] \end{aligned} \end{equation}$$

Setting the partial derivatives to zero yields the following simultaneous equations for the MLE:

$$\hat{\mu} = \frac{m \bar{x} \hat{\sigma}_y^2 + n \bar{y} \hat{\sigma}_x^2}{m \hat{\sigma}_y^2 + n \hat{\sigma}_x^2} \quad \quad \quad \hat{\sigma}_x^2 = \frac{1}{m} \sum_{i=1}^m (x_i - \hat{\mu})^2 \quad \quad \quad \hat{\sigma}_y^2 = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{\mu})^2.$$

These equations give us the conditional MLEs for each of the parameters, when the other parameters are known. To find the unconditional MLEs for each of our parameters we need to solve these simultaneous equations. This is a large algebraic exercise, which I will leave to you.

---

Solving via profile log-likelihood: Rather than solving these simultaneous equations directly, we can go back and substitute the form of the MLEs for the variance parameters back into the original log-likelihood function to obtain the profile log-likelihood:

$$\begin{equation} \begin{aligned} \ell_*(\mu) \equiv \ell(\mu,\hat{\sigma}_x,\hat{\sigma}_y) &= - m \ln \hat{\sigma}_x - n \ln \hat{\sigma}_y -\frac{1}{2} \Bigg[ \sum_{i=1}^m \frac{(x_i - \mu)^2}{\hat{\sigma}_x^2} + \sum_{i=1}^n \frac{(y_i - \mu)^2}{\hat{\sigma}_y^2} \Bigg] \\[6pt] &= - \frac{m}{2} \cdot \ln \Big( \sum_{i=1}^m (x_i - \mu)^2 \Big) - \frac{n}{2} \cdot \ln \Big( \sum_{i=1}^n (y_i - \mu)^2 \Big) + \text{const}. \\[6pt] \end{aligned} \end{equation}$$

The corresponding score function is:

$$\frac{d\ell_*}{d\mu}(\mu) = \frac{m^2(\bar{x} - \mu)}{\sum_{i=1}^m (x_i - \mu)^2} + \frac{n^2 (\bar{y} - \mu)}{\sum_{i=1}^n (y_i - \mu)^2}.$$

Setting this function to zero yields the following cubic equation for the critical points:

$$0 = m^2 (\bar{x}-\hat{\mu}) \sum_{i=1}^n (y_i - \hat{\mu})^2 + n^2 (\bar{y}-\hat{\mu}) \sum_{i=1}^m (x_i - \hat{\mu})^2.$$

It should be possible to find a unique maximising critical point that gives the MLE. (Substitute into the above conditional MLE equations as a check on your working.) Again, this is a large algebraic exercise that I will leave to you.

---