2
$\begingroup$

In maximum likelihood estimation for the parameter, mean $\mu$ of multivariate normal distribution, I end up with the following result:

$\mu_{MLE} = \dfrac{1}{n}\sum_{i=1}^n x_i$

The way I understand the above equation is the following way below, but I am wondering if the above equation means the following equation:

$\mu_{MLE} = \begin{bmatrix}\mu_1\\\mu_2\\\vdots\\\mu_n\end{bmatrix} = \begin{bmatrix}\dfrac{1}{n}\sum_{i=1}^n x_1\\\dfrac{1}{n}\sum_{i=1}^n x_2\\\vdots\\\dfrac{1}{n}\sum_{i=1}^n x_i\end{bmatrix} = \begin{bmatrix}E(X_1)\\E(X_2)\\\vdots\\E(X_n)\end{bmatrix} = \begin{bmatrix}\bar{X_1} \\ \bar{X_2} \\ \vdots \\ \bar{X_n}\end{bmatrix}$

And also Maximum Likelhood Estimation for $\Sigma$, I have ended up with the following equation:

$\Sigma_{MLE} = \dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T = \begin{bmatrix}Var(\sigma_{11}) & Cov(\sigma_{12}) & \dots & Cov(\sigma_{1n}) \\ Cov(\sigma_{21}) & Var(\sigma_{22}) & \dots & Cov(\sigma_{2n}) \\ {}& \vdots& \\ Cov(\sigma_{n1}) & Cov(\sigma_{n2}) & \dots & Var(\sigma_{nn}) \end{bmatrix}$

Somehow, I got the equation following some algebraic steps, but I can't see how this part, $\dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T$, works and produces the covariance matrix. Hope to hear some explanations.

$\endgroup$
7
$\begingroup$

It's just linear algebra. Let's take a concrete example where $n=3$. Then ${\bf \mu} \in \mathbb{R}^{3 \times 1}$.

$$\Sigma_{MLE} = \dfrac{1}{3}\sum_{i=1}^3(x_i-{\bf \mu})(x_i-{\bf \mu})^T = \frac13 \left( \begin{array}{c} x_{1} - \mu \\ x_{2} - \mu \\ x_{3} - \mu\end{array} \right)\left( \begin{array}{ccc} x_{1} - \mu & x_{2} - \mu & x_{3} - \mu\end{array} \right) $$ $$ = \frac13 \left( \begin{array}{ccc} (x_{1} - \mu)^2 & (x_{1} - \mu)(x_{2} - \mu) & (x_{1} - \mu)(x_{3} - \mu) \\ (x_{2} - \mu)(x_{1} - \mu) & (x_{2} - \mu)^2 & (x_{2} - \mu)(x_{3} - \mu) \\ (x_{3} - \mu)(x_{1} - \mu) & (x_{3} - \mu)(x_{2} - \mu) & (x_{3} - \mu)^2 \end{array} \right) $$ $$ = \left( \begin{array}{ccc} Var(x_1) & Cov(x_1,x_2) & Cov(x_1,x_3) \\ Cov(x_2,x_1) & Var(x_2) & Cov(x_2,x_3)\\ Cov(x_3,x_1) & Cov(x_3,x_2) & Var(x_3) \end{array} \right) $$

Also note that $Var(\sigma_{11})$ is odd notation. You might be thinking $\sigma^2_{x_1}$ or $Var(x_1)$.

$\endgroup$
  • $\begingroup$ Excellent ! So simple yet so clear - I wished I had seen this answer a while ago $\endgroup$ – Xavier Bourret Sicotte Jun 14 '18 at 16:55
  • $\begingroup$ Glad I could help! $\endgroup$ – ilanman Jun 14 '18 at 18:20
3
$\begingroup$

I can't see how this part, $\dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T$, works and produces the covariance matrix.

I think it is mostly the mathematical notation that confuses you. I, myself, interpret in this notation system:

$\Sigma_{MLE} =: E[(X-\mu)(X-\mu)^T] = \frac1n \left( \begin{array}{c} x_{1} - \mu \\ x_{2} - \mu \\ x_{3} - \mu \\ ...\\ x_{n} - \mu \end{array} \right)\left( \begin{array}{ccc} x_{1} - \mu & x_{2} - \mu & x_{3} - \mu & ...& x_{n} - \mu\end{array} \right)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.