# Maximum Likelihood Estimators - Multivariate Gaussian

### Context

The Multivariate Gaussian appears frequently in Machine Learning and the following results are used in many ML books and courses without the derivations.

Given data in form of a matrix $\mathbf{X}$ of dimensions $m \times p$, if we assume that the data follows a $p$-variate Gaussian distribution with parameters mean $\mu$ ( $p \times 1$) and covariance matrix $\Sigma$ ($p \times p$) the Maximum Likelihood Estimators are given by:

• $\hat \mu = \frac{1}{m} \sum_{i=1}^m \mathbf{ x^{(i)} } = \mathbf{\bar{x}}$
• $\hat \Sigma = \frac{1}{m} \sum_{i=1}^m \mathbf{(x^{(i)} - \hat \mu) (x^{(i)} -\hat \mu)}^T$

I understand that knowledge of the multivariate Gaussian is a pre-requisite for many ML courses, but it would be helpful to have the full derivation in a self contained answer once and for all as I feel many self-learners are bouncing around the stats.stackexchange and math.stackexchange websites looking for answers.

### Question

What is the full derivation of the Maximum Likelihood Estimators for the multivariate Gaussian

### Examples:

These lecture notes (page 11) on Linear Discriminant Analysis, or these ones make use of the results and assume previous knowledge.

There are also a few posts which are partly answered or closed:

### Deriving the Maximum Likelihood Estimators

Assume that we have $$m$$ random vectors, each of size $$p$$: $$\mathbf{X^{(1)}, X^{(2)}, \dotsc, X^{(m)}}$$ where each random vectors can be interpreted as an observation (data point) across $$p$$ variables. If each $$\mathbf{X}^{(i)}$$ are i.i.d. as multivariate Gaussian vectors:

$$\mathbf{X^{(i)}} \sim \mathcal{N}_p(\mu, \Sigma)$$

Where the parameters $$\mu, \Sigma$$ are unknown. To obtain their estimate we can use the method of maximum likelihood and maximize the log likelihood function.

Note that by the independence of the random vectors, the joint density of the data $$\mathbf{ \{X^{(i)}}, i = 1,2, \dotsc ,m\}$$ is the product of the individual densities, that is $$\prod_{i=1}^m f_{\mathbf{X^{(i)}}}(\mathbf{x^{(i)} ; \mu , \Sigma })$$. Taking the logarithm gives the log-likelihood function

\begin{aligned} l(\mathbf{ \mu, \Sigma | x^{(i)} }) & = \log \prod_{i=1}^m f_{\mathbf{X^{(i)}}}(\mathbf{x^{(i)} | \mu , \Sigma }) \\ & = \log \ \prod_{i=1}^m \frac{1}{(2 \pi)^{p/2} |\Sigma|^{1/2}} \exp \left( - \frac{1}{2} \mathbf{(x^{(i)} - \mu)^T \Sigma^{-1} (x^{(i)} - \mu) } \right) \\ & = \sum_{i=1}^m \left( - \frac{p}{2} \log (2 \pi) - \frac{1}{2} \log |\Sigma| - \frac{1}{2} \mathbf{(x^{(i)} - \mu)^T \Sigma^{-1} (x^{(i)} - \mu) } \right) \end{aligned}

\begin{aligned} l(\mu, \Sigma ; ) & = - \frac{mp}{2} \log (2 \pi) - \frac{m}{2} \log |\Sigma| - \frac{1}{2} \sum_{i=1}^m \mathbf{(x^{(i)} - \mu)^T \Sigma^{-1} (x^{(i)} - \mu) } \end{aligned}

### Deriving $$\hat \mu$$

To take the derivative with respect to $$\mu$$ and equate to zero we will make use of the following matrix calculus identity:

$$\mathbf{ \frac{\partial w^T A w}{\partial w} = 2Aw}$$ if $$\mathbf{w}$$ does not depend on $$\mathbf{A}$$ and $$\mathbf{A}$$ is symmetric.

\begin{aligned} \frac{\partial }{\partial \mu} l(\mathbf{ \mu, \Sigma | x^{(i)} }) & = \sum_{i=1}^m \mathbf{ \Sigma^{-1} ( x^{(i)} - \mu ) } = 0 \\ & \text{Since \Sigma is positive definite} \\ 0 & = m \mu - \sum_{i=1}^m \mathbf{ x^{(i)} } \\ \hat \mu &= \frac{1}{m} \sum_{i=1}^m \mathbf{ x^{(i)} } = \mathbf{\bar{x}} \end{aligned}

Which is often called the sample mean vector.

### Deriving $$\hat \Sigma$$

Deriving the MLE for the covariance matrix requires more work and the use of the following linear algebra and calculus properties:

• The trace is invariant under cyclic permutations of matrix products: $$\mathrm{tr}\left[ABC\right] = \mathrm{tr}\left[CAB\right] = \mathrm{tr}\left[BCA\right]$$
• Since $$x^TAx$$ is scalar, we can take its trace and obtain the same value: $$x^TAx = \mathrm{tr}\left[x^TAx\right] = \mathrm{tr}\left[xx^TA\right]$$
• $$\frac{\partial}{\partial A} \mathrm{tr}\left[AB\right] = B^T$$
• $$\frac{\partial}{\partial A} \log |A| = (A^{-1})^T = (A^T)^{-1}$$
• The determinant of the inverse of an invertible matrix is the inverse of the determinant: $$|A| = \frac{1}{|A^{-1}|}$$

Combining these properties allows us to calculate

$$\frac{\partial}{\partial A} x^TAx =\frac{\partial}{\partial A} \mathrm{tr}\left[xx^TA\right] = [xx^T]^T = \left(x^{T}\right)^Tx^T = xx^T$$

Which is the outer product of the vector $$x$$ with itself.

We can now re-write the log-likelihood function and compute the derivative w.r.t. $$\Sigma^{-1}$$ (note $$C$$ is constant)

\begin{aligned} l(\mathbf{ \mu, \Sigma | x^{(i)} }) & = \text{C} - \frac{m}{2} \log |\Sigma| - \frac{1}{2} \sum_{i=1}^m \mathbf{(x^{(i)} - \mu)^T \Sigma^{-1} (x^{(i)} - \mu) } \\ & = \text{C} + \frac{m}{2} \log |\Sigma^{-1}| - \frac{1}{2} \sum_{i=1}^m \mathrm{tr}\left[ \mathbf{(x^{(i)} - \mu) (x^{(i)} - \mu)^T \Sigma^{-1} } \right] \\ \frac{\partial }{\partial \Sigma^{-1}} l(\mathbf{ \mu, \Sigma | x^{(i)} }) & = \frac{m}{2} \Sigma - \frac{1}{2} \sum_{i=1}^m \mathbf{(x^{(i)} - \mu) (x^{(i)} - \mu)}^T \ \ \text{Since \Sigma^T = \Sigma} \end{aligned}

Equating to zero and solving for $$\Sigma$$

\begin{aligned} 0 &= m \Sigma - \sum_{i=1}^m \mathbf{(x^{(i)} - \mu) (x^{(i)} - \mu)}^T \\ \hat \Sigma & = \frac{1}{m} \sum_{i=1}^m \mathbf{(x^{(i)} - \hat \mu) (x^{(i)} -\hat \mu)}^T \end{aligned}

### Sources

• In the derivation for $\mu$, why does $\Sigma$ need to be positive definite? Does it seem enough that $\Sigma$ is invertible? For an invertible matrix $A$, $Ax=0$ only when $x=0$? Apr 14, 2019 at 3:01
• To clarify, $\Sigma$ is an $m \times m$ matrix that may have finite diagonal and non-diagonal components indicating correlation between vectors, correct? If that is the case, in what sense are these vectors independent? Also, why is the joint probability function equal to the likelihood? Shouldn't the joint density, $f(x,y)$, be equal to the likelihood multiplied by the prior, i.e. $f(x|y)f(y)$? Apr 25, 2019 at 2:07
• @TomBennett the sigma matrix is positive definite by definition - see stats.stackexchange.com/questions/52976/… for the proof. The matrix calculus identity requires the matrix to be symmetric, not positive definite. But since positive definite matrices are always symmetric that works Apr 26, 2019 at 0:24
• Yes indeed - independence between observations allow to get the likelihood - the wording may be unclear faie enough - this is the multivariate version of the likelihood. The prior is still irrelevant regardless May 4, 2019 at 18:16
• @Tobbey - yes indeed, feel free to post another answer below and I will upvote it Sep 19, 2022 at 10:14

An alternate proof for $$\widehat{\Sigma}$$ that takes the derivative with respect to $$\Sigma$$ directly:

Picking up with the log-likelihood as above: $$\begin{eqnarray} \ell(\mu, \Sigma) &=& C - \frac{m}{2}\log|\Sigma|-\frac{1}{2} \sum_{i=1}^m \text{tr}\left[(\mathbf{x}^{(i)}-\mu)^T \Sigma^{-1} (\mathbf{x}^{(i)}-\mu)\right]\\ &=&C - \frac{1}{2}\left(m\log|\Sigma| + \sum_{i=1}^m\text{tr} \left[(\mathbf{x}^{(i)}-\mu)(\mathbf{x}^{(i)}-\mu)^T\Sigma^{-1} \right]\right)\\ &=&C - \frac{1}{2}\left(m\log|\Sigma| +\text{tr}\left[ S_\mu \Sigma^{-1} \right] \right) \end{eqnarray}$$ where $$S_\mu = \sum_{i=1}^m (\mathbf{x}^{(i)}-\mu)(\mathbf{x}^{(i)}-\mu)^T$$ and we have used the cyclic and linear properties of $$\text{tr}$$. To compute $$\partial \ell /\partial \Sigma$$ we first observe that $$\frac{\partial}{\partial \Sigma} \log |\Sigma| = \Sigma^{-T}=\Sigma^{-1}$$ by the fourth property above. To take the derivative of the second term we will need the property that $$\frac{\partial}{\partial X}\text{tr}\left( A X^{-1} B\right) = -(X^{-1}BAX^{-1})^T.$$ (from The Matrix Cookbook, equation 63). Applying this with $$B=I$$ we obtain that $$\frac{\partial}{\partial \Sigma}\text{tr}\left[S_\mu \Sigma^{-1}\right] = -\left( \Sigma^{-1} S_\mu \Sigma^{-1}\right)^T = -\Sigma^{-1} S_\mu \Sigma^{-1}$$ because both $$\Sigma$$ and $$S_\mu$$ are symmetric. Then $$\frac{\partial}{\partial \Sigma}\ell(\mu, \Sigma) \propto m \Sigma^{-1} - \Sigma^{-1} S_\mu \Sigma^{-1}.$$ Setting this to 0 and rearranging gives $$\widehat{\Sigma} = \frac{1}{m}S_\mu.$$

This approach is more work than the standard one using derivatives with respect to $$\Lambda = \Sigma^{-1}$$, and requires a more complicated trace identity. I only found it useful because I currently need to take derivatives of a modified likelihood function for which it seems much harder to use $$\partial/{\partial \Sigma^{-1}}$$ than $$\partial/\partial \Sigma$$.

While previous answers are correct, mentioning the trace is unnecessary (from a personal point of view).

The following derivation might be more succinct:

• isnt this exactly the same as my answer, but skipping the steps explaining the derivative of the matrix ? Apologies but I fail to see what this is adding Oct 29, 2021 at 15:44
• Personally, I find proofs (in my textbooks) with trace messing up with my ability to remember/understand/appreciate the proof. It's like proving another theorem (2 in my answer) every time, since 2 in my answer is standard results in Matrix reference book, as I listed. This is just for people who might have the same issue. Anyway, all best intentions!
– Wiza
Oct 30, 2021 at 4:14