The question I am trying to answer is confusing me as I don't know where to start to find a likelihood estimation

The Question

$y_i = μ + e_i $

where the $e_i$ are independent variables distributed $N(0, σ_i^2)$

What would the maximum likelihood estimator of $μ$ be? And also, what would the mean, variance and distribution of the MLE be?

My attempt: I have tried to find the expectation of $y_i$, giving an answer of μ as the $e_i$ have mean 0. Without a distribution (i.e. Binomial, Normal) for the model, how do you find the MLE?

  • $\begingroup$ Please add the [self-study] tag & read its wiki. $\endgroup$ Nov 30 '14 at 15:32
  • 1
    $\begingroup$ Can you please verify that your error terms do really vary with i. Because you seem to be new to MLE and as a starting point usually iid random variables are used, i. e. the variance is equal for all observations. $\endgroup$
    – random_guy
    Nov 30 '14 at 16:07
  • $\begingroup$ I've done MLE with iid rvs, which is fairly easy to work through, but I've never had to work a multivariate normal estimator. As for the error terms, they do vary with i. Thanks for the help $\endgroup$
    – amse
    Nov 30 '14 at 16:12
  • 1
    $\begingroup$ And I think this is not even necessary because $\mu$ is equal for $y_i$ and they are independent. So, you could replace all matrix and vectors by using sums. $\endgroup$
    – random_guy
    Nov 30 '14 at 16:26
  • 1
    $\begingroup$ This makes little sense because you are attempting to estimate more parameters (one mean and $n$ variances) than you have data ($n$ altogether). $\endgroup$
    – whuber
    Dec 1 '14 at 1:04

In response to @Xi'an answer, let's assume that we have a sample size of $n=2$ with $y_1 \neq y_2$. Then

$$\sum_{i=1}^2 (y_i-\hat\mu)^{-1}=0 \Rightarrow \frac {1}{y_1-\hat \mu} + \frac {1}{y_2-\hat \mu} = 0 \Rightarrow \frac {y_2 - \hat \mu + y_1 - \hat \mu}{(y_1-\hat \mu)(y_2-\hat \mu)}= 0$$

$$\Rightarrow y_1 - \hat \mu + y_2 - \hat \mu = 0 \Rightarrow \hat \mu = \frac {y_1 + y_2}{2}$$

Plugging this into the derived first-order conditions for the variances, we obtain

$$\hat\sigma_1^2 = \left(y_1-\frac {y_1 + y_2}{2}\right)^2 = \frac {(y_1-y_2)^2}{4}$$


$$\hat\sigma_2^2 = \left(y_2-\frac {y_1 + y_2}{2}\right)^2 = \frac {(y_1-y_2)^2}{4}$$

Hmmm, so for any sample of size $n=2$, we will be estimating the two different variances as equal... Then why go into the whole trouble and treat them as different in the first place? Does this generalize in some sense for bigger $n$? Does this have any relation with the fact that we are estimating $n+1$ unknown parameters having only $n$ data points?

  • $\begingroup$ Sorry to be dense, but how does "$\sum_{i=1}^2 (y_i-\hat\mu)^{-1}=0 \Rightarrow y_1 - \hat \mu + y_2 - \hat \mu = 0$" follow? $\endgroup$
    – Glen_b
    Dec 4 '14 at 0:16
  • 1
    $\begingroup$ @Glen_b Added the calculation steps. $\endgroup$ Dec 4 '14 at 0:51
  • 1
    $\begingroup$ Yeah, I was being dense. Now that I have stopped mentally taking absolute values of each term in the original equation it makes more sense there. $\endgroup$
    – Glen_b
    Dec 4 '14 at 1:03
  • $\begingroup$ This does not answer the original question. $\endgroup$
    – Xi'an
    Dec 9 '14 at 12:55
  • 1
    $\begingroup$ @Xi'an No, but it brings into the surface some aspects of your answer, and it says so explicitly, in its first-first sentence. It is not the first time such a "dialogue" happens in CV. $\endgroup$ Dec 9 '14 at 13:09

To find the maximum likelihood a generic approach is to write down the likelihood function as a starting point.

In this example, $Y_i\sim\mathcal{N}(\mu,\sigma_i^2)$ means that $$ L(\mu,\sigma_1,\ldots,\sigma_n) = \prod_{i=1}^n \dfrac{\exp\{-(y_i-\mu)^2/2\sigma_i^2\}}{\sqrt{2\pi}\sigma_i} $$ or $$ \ell(\mu,\sigma_1,\ldots,\sigma_n) =\log L(\mu,\sigma_1,\ldots,\sigma_n) = \sum_{i=1}^n -(y_i-\mu)^2/2\sigma_i^2 - \log \sigma_i + \text{cst} $$

Now that you have your (log-)likelihood function, you have to find its maximum in $(\mu,\sigma_1,\ldots,\sigma_n)$. A standard approach at this stage is to look at the derivatives against every component of the parameter: \begin{align*} \frac{\partial \ell}{\partial \mu}(\mu,\sigma_1,\ldots,\sigma_n) &= \sum_{i=1}^n (y_i-\mu)/\sigma_i^2 \\ \frac{\partial \ell}{\partial \sigma_i} (\mu,\sigma_1,\ldots,\sigma_n) &= (y_i-\mu)^2/\sigma_i^3 - 1/\sigma_i \qquad i=1,\ldots,n \end{align*} If you look at the solutions $(\hat\mu,\hat\sigma_1,\ldots,\hat\sigma_n)$ of $$ \frac{\partial \ell}{\partial \mu}(\mu,\sigma_1,\ldots,\sigma_n) =0,\qquad \frac{\partial \ell}{\partial \sigma_i}(\mu,\sigma_1,\ldots,\sigma_n) =0,\qquad i=1,\ldots,n, $$ you find \begin{align*} \sum_{i=1}^n \hat\sigma_i^{-2} y_i - n \hat\mu \sum_{i=1}^n \hat\sigma_i^{-2} &= 0\\ (y_i-\hat\mu)^2/\hat\sigma_i^3 - 1/\hat\sigma_i &= 0 \qquad i=1,\ldots,n \end{align*} which leads to $$ \hat\sigma_i^2 = (y_i-\hat\mu)^2 \qquad i=1,\ldots,n $$ hence to $$ \sum_{i=1}^n (y_i-\hat\mu)/\hat\sigma_i^2 = \sum_{i=1}^n (y_i-\hat\mu)^{-1}=0 $$ This equation does not have a closed form expression in $\hat\mu$.

This is the generic way to derive the likelihood equations and their solution(s). However, in some cases like the current one, the maximum of the likelihood function might occur at the boundary of the parameter space. Take indeed $\mu=y_1$. Then the log-likelihood function is equal to $$ -\log(\sigma_1)-\sum_{i=2}^n (y_i-\mu)^2/2\sigma_i^2 -\sum_{i=2}^n \log \sigma_i + \text{cst} $$ and if we let $\sigma_1$ go to zero, the log-likelihood goes to infinity. The same property holds for $\mu=y_2,\ldots,y_n$. The maximum of the likelihood is thus at the boundary of the parameter space $$ \mathbb{R}\times\mathbb{R}^*_+\times\cdots\times\mathbb{R}^*_+ $$ (excluding $0$ as possible value for the $\sigma_i$) and therefore there is no maximum likelihood estimator.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.