Need help to understand Maximum Likelihood Estimation for multivariate normal distribution?

Question

In maximum likelihood estimation for the parameter, mean $\mu$ of multivariate normal distribution, I end up with the following result:

$\mu_{MLE} = \dfrac{1}{n}\sum_{i=1}^n x_i$

The way I understand the above equation is the following way below, but I am wondering if the above equation means the following equation:

$\mu_{MLE} = \begin{bmatrix}\mu_1\\\mu_2\\\vdots\\\mu_n\end{bmatrix} = \begin{bmatrix}\dfrac{1}{n}\sum_{i=1}^n x_1\\\dfrac{1}{n}\sum_{i=1}^n x_2\\\vdots\\\dfrac{1}{n}\sum_{i=1}^n x_i\end{bmatrix} = \begin{bmatrix}E(X_1)\\E(X_2)\\\vdots\\E(X_n)\end{bmatrix} = \begin{bmatrix}\bar{X_1} \\ \bar{X_2} \\ \vdots \\ \bar{X_n}\end{bmatrix}$

And also Maximum Likelhood Estimation for $\Sigma$, I have ended up with the following equation:

$\Sigma_{MLE} = \dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T = \begin{bmatrix}Var(\sigma_{11}) & Cov(\sigma_{12}) & \dots & Cov(\sigma_{1n}) \\ Cov(\sigma_{21}) & Var(\sigma_{22}) & \dots & Cov(\sigma_{2n}) \\ {}& \vdots& \\ Cov(\sigma_{n1}) & Cov(\sigma_{n2}) & \dots & Var(\sigma_{nn}) \end{bmatrix}$

Somehow, I got the equation following some algebraic steps, but I can't see how this part, $\dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T$, works and produces the covariance matrix. Hope to hear some explanations.

Answer 1

It's just linear algebra. Let's take a concrete example where $n=3$. Then ${\bf \mu} \in \mathbb{R}^{3 \times 1}$.

$$\Sigma_{MLE} = \dfrac{1}{3}\sum_{i=1}^3(x_i-{\bf \mu})(x_i-{\bf \mu})^T = \frac13 \left( \begin{array}{c} x_{1} - \mu \\ x_{2} - \mu \\ x_{3} - \mu\end{array} \right)\left( \begin{array}{ccc} x_{1} - \mu & x_{2} - \mu & x_{3} - \mu\end{array} \right) $$ $$ = \frac13 \left( \begin{array}{ccc} (x_{1} - \mu)^2 & (x_{1} - \mu)(x_{2} - \mu) & (x_{1} - \mu)(x_{3} - \mu) \\ (x_{2} - \mu)(x_{1} - \mu) & (x_{2} - \mu)^2 & (x_{2} - \mu)(x_{3} - \mu) \\ (x_{3} - \mu)(x_{1} - \mu) & (x_{3} - \mu)(x_{2} - \mu) & (x_{3} - \mu)^2 \end{array} \right) $$ $$ = \left( \begin{array}{ccc} Var(x_1) & Cov(x_1,x_2) & Cov(x_1,x_3) \\ Cov(x_2,x_1) & Var(x_2) & Cov(x_2,x_3)\\ Cov(x_3,x_1) & Cov(x_3,x_2) & Var(x_3) \end{array} \right) $$

Also note that $Var(\sigma_{11})$ is odd notation. You might be thinking $\sigma^2_{x_1}$ or $Var(x_1)$.

Answer 2

I can't see how this part, $\dfrac{1}{n}\sum_{i=1}^n(x_i-\mu)(x_i-\mu)^T$, works and produces the covariance matrix.

I think it is mostly the mathematical notation that confuses you. I, myself, interpret in this notation system:

$\Sigma_{MLE} =: E[(X-\mu)(X-\mu)^T] = \frac1n \left( \begin{array}{c} x_{1} - \mu \\ x_{2} - \mu \\ x_{3} - \mu \\ ...\\ x_{n} - \mu \end{array} \right)\left( \begin{array}{ccc} x_{1} - \mu & x_{2} - \mu & x_{3} - \mu & ...& x_{n} - \mu\end{array} \right)$