3
$\begingroup$

My problem is as follows,

Find the maximum likelihood estimator, $\mu^{MLE}$ from $(m+n)$ samples, where $X_1, \cdots, X_m \sim N(\mu, \sigma_1^2), Y_1, \cdots, Y_n \sim N(\mu, \sigma_2^2),$

where $\sigma_1^2, \sigma_2^2$ : both unknown.

My attempt

$$ \begin{aligned} \log L(\mu) = l(\mu) &= \mathrm{constant}-\frac{1}{2} \left[\sum_{i=1}^m\frac{(x_i - \mu)^2}{\sigma_1^2} + \sum_{j=1}^n\frac{(y_j - \mu)^2}{\sigma_2^2} \right] \\ \frac{\partial l(\mu)}{\partial \mu} &= - \frac{\mu}{\sigma_1^2} \sum_{i=1}^m x_i^2 - \frac{\mu}{\sigma_2^2} \sum_{j=1}^n y_j^2 + \frac{m \bar{x}}{\sigma_1^2} + \frac{n \bar{y}}{\sigma_2^2} \end{aligned} $$

This yields $\hat{\mu^{MLE}} =\frac{ m \bar{x}/\sigma_1^2 +n\bar{y}/\sigma_2^2 }{m /\sigma_1^2 +n/\sigma_2^2}$, but the main problem is, I basically don't know the variances. Any help will be appreciated.

$\endgroup$
  • 3
    $\begingroup$ Hi: get rid of the variances right at the beginning by using their respective MLE's. this way, you just have one unknown parameter to solve for. This is called the profiled ( the term concentrated is sometimes used ) likelihood and is often helpful when maximizing a likelihood when there is more than one parameter. I don' know the specific answer but that should help. $\endgroup$ – mlofton Oct 9 '18 at 11:23
  • 2
    $\begingroup$ One other thing: Your answer should come out to some weighted function of the two sets of observations where the observations are weighted by the inverse of their variance. This way, larger variance observations get less weight. $\endgroup$ – mlofton Oct 9 '18 at 11:26
  • $\begingroup$ @mlofton Even though I adopt such technique, how can I estimate confidence interval for $\mu$? $\endgroup$ – moreblue Oct 9 '18 at 11:44
  • 1
    $\begingroup$ I may not be able to derive it regardless, but, to calculate the confidence interval, one needs the expression for the estimate of $\mu$ and it's standard deviation. So, the MLE expression is needed to obtain it's standard deviation. Soiunds like you derived it so send the exact expression. $\endgroup$ – mlofton Oct 9 '18 at 14:45
  • $\begingroup$ Actually, there are two (independent) samples - $(X_1,X_2,\ldots,X_n)$ and $(Y_1,Y_2,\ldots,Y_m)$. The sample size is $m+n$. $\endgroup$ – StubbornAtom Oct 9 '18 at 19:32
2
$\begingroup$

Your attempt at the problem is correct so far, but you have only derived the conditional MLE when the variance parameters are known. To derive the unconditional MLE for the mean parameter, you will need to derive the corresponding equations for the MLEs of the variance parameters and then solve the resulting set of simultaneous equations. This should give you a unique MLE estimating each of the three parameters in the model. Let me show you how to do this.


Derivation of the full MLE: For greater clarity, I will denote the variance parameters as $\sigma_x^2$ and $\sigma_y^2$ rather than denoting them with number subscripts. From your specified model, the log-likelihood for your observed data (ignoring an additive constant) can be written as:

$$\begin{equation} \begin{aligned} \ell(\mu,\sigma_x,\sigma_y) &= - m \ln \sigma_x - n \ln \sigma_y -\frac{1}{2} \Bigg[ \sum_{i=1}^m \frac{(x_i - \mu)^2}{\sigma_x^2} + \sum_{i=1}^n \frac{(y_i - \mu)^2}{\sigma_y^2} \Bigg] \\[6pt] &= - m \ln \sigma_x - n \ln \sigma_y -\frac{1}{2} \Bigg[ \frac{1}{\sigma_x^2} \sum_{i=1}^m (x_i^2 - 2 \mu x_i + \mu^2) + \frac{1}{\sigma_y^2} \sum_{i=1}^n (y_i^2 - 2 \mu y_i + \mu^2) \Bigg] \\[6pt] &= - m \ln \sigma_x - n \ln \sigma_y -\frac{1}{2} \Bigg( \frac{1}{\sigma_x^2} \sum_{i=1}^m x_i^2 + \frac{1}{\sigma_y^2} \sum_{i=1}^n y_i^2 \Bigg) \\[6pt] &\quad \quad \quad \quad \quad \quad \quad \text{ } \text{ } + \Bigg( \frac{m\bar{x}}{\sigma_x^2} + \frac{n\bar{y}}{\sigma_y^2} \Bigg) \mu -\frac{1}{2} \Bigg( \frac{m}{\sigma_x^2} + \frac{n}{\sigma_y^2} \Bigg) \mu^2. \\[6pt] \end{aligned} \end{equation}$$

where $\bar{x} = \sum_{i=1}^m x_i / m$ and $\bar{y} = \sum_{i=1}^n y_i / n$ are the sample means of the parts. Hence, your score function consists of the following partial derivatives:

$$\begin{equation} \begin{aligned} \frac{\partial \ell}{\partial \mu}(\mu,\sigma_x,\sigma_y) &= \Bigg( \frac{m\bar{x}}{\sigma_x^2} + \frac{n\bar{y}}{\sigma_y^2} \Bigg) - \Bigg( \frac{m}{\sigma_x^2} + \frac{n}{\sigma_y^2} \Bigg) \mu, \\[10pt] \frac{\partial \ell}{\partial \sigma_x}(\mu,\sigma_x,\sigma_y) &= - \frac{1}{\sigma_x^3} \Bigg( m \sigma_x^2 - \sum_{i=1}^m (x_i - \mu)^2 \Bigg) , \\[10pt] \frac{\partial \ell}{\partial \sigma_y}(\mu,\sigma_x,\sigma_y) &= - \frac{1}{\sigma_y^3} \Bigg( n \sigma_y^2 - \sum_{i=1}^n (y_i - \mu)^2 \Bigg) . \\[10pt] \end{aligned} \end{equation}$$

Setting the partial derivatives to zero yields the following simultaneous equations for the MLE:

$$\hat{\mu} = \frac{m \bar{x} \hat{\sigma}_y^2 + n \bar{y} \hat{\sigma}_x^2}{m \hat{\sigma}_y^2 + n \hat{\sigma}_x^2} \quad \quad \quad \hat{\sigma}_x^2 = \frac{1}{m} \sum_{i=1}^m (x_i - \hat{\mu})^2 \quad \quad \quad \hat{\sigma}_y^2 = \frac{1}{n} \sum_{i=1}^n (y_i - \hat{\mu})^2.$$

These equations give us the conditional MLEs for each of the parameters, when the other parameters are known. To find the unconditional MLEs for each of our parameters we need to solve these simultaneous equations. This is a large algebraic exercise, which I will leave to you.


Solving via profile log-likelihood: Rather than solving these simultaneous equations directly, we can go back and substitute the form of the MLEs for the variance parameters back into the original log-likelihood function to obtain the profile log-likelihood:

$$\begin{equation} \begin{aligned} \ell_*(\mu) \equiv \ell(\mu,\hat{\sigma}_x,\hat{\sigma}_y) &= - m \ln \hat{\sigma}_x - n \ln \hat{\sigma}_y -\frac{1}{2} \Bigg[ \sum_{i=1}^m \frac{(x_i - \mu)^2}{\hat{\sigma}_x^2} + \sum_{i=1}^n \frac{(y_i - \mu)^2}{\hat{\sigma}_y^2} \Bigg] \\[6pt] &= - \frac{m}{2} \cdot \ln \Big( \sum_{i=1}^m (x_i - \mu)^2 \Big) - \frac{n}{2} \cdot \ln \Big( \sum_{i=1}^n (y_i - \mu)^2 \Big) + \text{const}. \\[6pt] \end{aligned} \end{equation}$$

The corresponding score function is:

$$\frac{d\ell_*}{d\mu}(\mu) = \frac{m^2(\bar{x} - \mu)}{\sum_{i=1}^m (x_i - \mu)^2} + \frac{n^2 (\bar{y} - \mu)}{\sum_{i=1}^n (y_i - \mu)^2}.$$

Setting this function to zero yields the following cubic equation for the critical points:

$$0 = m^2 (\bar{x}-\hat{\mu}) \sum_{i=1}^n (y_i - \hat{\mu})^2 + n^2 (\bar{y}-\hat{\mu}) \sum_{i=1}^m (x_i - \hat{\mu})^2.$$

It should be possible to find a unique maximising critical point that gives the MLE. (Substitute into the above conditional MLE equations as a check on your working.) Again, this is a large algebraic exercise that I will leave to you.


$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.