Why do we use the determinant of the covariance matrix when using the multivariate normal?

Question

I am not well versed in statistics. I wanted to know why we use the determinant of the covariance matrix instead of having the covariance matrix itself when writing down the multivariate normal distribution. Why do we do this and what is the intuition behind it?

I have noticed that for a basic Maximum Lkikelihood estimation with normal errors can be written as follows for the univariate case:

$\frac{1}{\sqrt{2\pi \sigma^2}}\cdot \exp(-\frac{1}{2}(\epsilon/\sigma)^2)$

and as follows for the multivariate case:

$\frac{1}{\sqrt{2\pi |\Sigma|}}\cdot \exp(-\frac{1}{2}(\frac{\epsilon}{|\Sigma|})^2)$

What I want to know is why do we use the determinant of the covariance matrix in the multivariate case. In the univariate case we have the variance sigma^2, but in the multivariate case we write the determinant of the variance.

Answer 1

Instead of jumping to the multivariate case in matrix form, look at the bivariate case first:

Can you recognize the portion of the denominator that is the determinant of the variance-covariance matrix below?

In the univariate case you don't have a determinant because $\sum$ consists of just one term. You don't have another variable, so you don't need to take into account any interaction between them.

Answer 2

Here another practical way to feel more confident about the vector and matrix notation of the multivariate normal. How did the transformation to the multivariate case work and generate $\Sigma_\mathcal{E}$ and $\mathbf{(y-\mu)}'(\Sigma_\mathcal{E}^{-1})(\mathbf{y-\mu})$ in the new joint density? We will show this briefly.

Lets say we have a random vector $\mathbf{y}$ with three variables as in

$$\mathbf{y}=\begin{pmatrix} y_1 \\ y_2 \\ y_3\end{pmatrix}, \mathbb{E}[\mathbf{y}]=\begin{pmatrix} \mu_1 \\ \mu_2 \\ \mu_3\end{pmatrix}, \Sigma_\mathcal{E}= \begin{pmatrix} \sigma_1^2 & 0 & 0 \\ 0 & \sigma_2^2 & 0 \\ 0 & 0 & \sigma_3^2 \end{pmatrix}$$

Clearly when these random values appear we observe them simultaneously and therefore the joint probability of observing particular values of this random vector simultaneously is equal to

$$P(y_1)\cdot P(y_2)\cdot P(y_3)$$

Where $P(...)$ represents the (marginal) probability distribution of the random variables. Since we reason from mutually independent standard normal random variables, we implicitly assume independence between the variables. This is crucial because only then can we simply multiply the marginal probabilities to arrive at joint probabilities. The logic is much the same as throwing two $6$ dots with two dices at once, under natural independence the probability is $(1/6)(1/6)≈2.8\%$. For our values of $y_1,y_2$ and $y_3$ we can apply the univariate Normal PDF, that is

$$P(y_1)=\dfrac{1}{\sqrt{2\pi}\sigma_1}\exp\left(-\dfrac{(y_1-\mu_1)^2}{2\sigma_1^2}\right) $$ $$P(y_2)=\dfrac{1}{\sqrt{2\pi}\sigma_2}\exp\left(-\dfrac{(y_2-\mu_2)^2}{2\sigma_2^2}\right) $$ $$P(y_3)=\dfrac{1}{\sqrt{2\pi}\sigma_3}\exp\left(-\dfrac{(y_3-\mu_3)^2}{2\sigma_3^2}\right) $$

And hence if we multiply we get

$$P(y_1)\cdot P(y_2)\cdot P(y_3)=\left(\dfrac{1}{\sqrt{2\pi}\sigma_1}\right)\left(\dfrac{1}{\sqrt{2\pi}\sigma_2}\right)\left(\dfrac{1}{\sqrt{2\pi}\sigma_3}\right)\exp\left(-\dfrac{(y_1-\mu_1)^2}{2\sigma_1^2}-\dfrac{(y_2-\mu_2)^2}{2\sigma_2^2}-\dfrac{(y_3-\mu_3)^2}{2\sigma_3^2}\right)$$

$$=\left(\dfrac{1}{\sqrt{2\pi}^3\sigma_1\sigma_2 \sigma_3}\right)\exp\left(-\dfrac{1}{2}\left(\dfrac{(y_1-\mu_1)^2}{\sigma_1^2}+\dfrac{(y_2-\mu_2)^2}{\sigma_2^2}+\dfrac{(y_3-\mu_3)^2}{\sigma_3^2}\right)\right)$$

Now notice that

$$\sigma_1^2 \sigma_2^2 \sigma_3^2=\det(\Sigma_\mathcal{E})=\sigma_1^2\left| {\begin{array}{cc} \sigma_2^2 & 0 \\ 0 & \sigma_3^2 \end{array} } \right|=\sigma_1^2 \sigma_2^2 \sigma_3^2 $$

and notice that

$$\mathbf{(y-\mu)}'(\Sigma_\mathcal{E}^{-1})(\mathbf{y-\mu}) = \\= \begin{pmatrix} y_1-\mu_1 & y_2-\mu_2 & y_3-\mu_3 \\\end{pmatrix} \begin{pmatrix} \sigma_1^2 & 0 & 0 \\ 0 & \sigma_2^2 & 0 \\ 0 & 0 & \sigma_3^2 \end{pmatrix}^{-1}\begin{pmatrix} y_1-\mu_1 \\ y_2-\mu_2 \\ y_3-\mu_3 \\\end{pmatrix} $$

$$=\begin{pmatrix} y_1-\mu_1 & y_2-\mu_2 & y_3-\mu_3 \\\end{pmatrix} \begin{pmatrix} 1/\sigma_1^2 & 0 & 0 \\ 0 & 1/\sigma_2^2 & 0 \\ 0 & 0 & 1/\sigma_3^2 \end{pmatrix}\begin{pmatrix} y_1-\mu_1 \\ y_2-\mu_2 \\ y_3-\mu_3 \\\end{pmatrix} $$

$$ = \begin{pmatrix} (y_1-\mu_1)/\sigma_1^2 & (y_2-\mu_2)/\sigma_2^2 & (y_3-\mu_3)/\sigma_3^2 \\\end{pmatrix} \begin{pmatrix} y_1-\mu_1 \\ y_2-\mu_2 \\ y_3-\mu_3 \\\end{pmatrix}$$

$$= \left(\dfrac{(y_1-\mu_1)^2}{\sigma_1^2}+\dfrac{(y_2-\mu_2)^2}{\sigma_2^2}+\dfrac{(y_3-\mu_3)^2}{\sigma_3^2}\right)$$

Therefore we can write the multivariate normal (joint distribution) as

$$\dfrac{1}{\sqrt{(2\pi)^k\det(\Sigma_\mathcal{E})}}\exp\left(-\dfrac{\mathbf{(y-\mu)}'(\Sigma_\mathcal{E}^{-1})(\mathbf{y-\mu})}{2}\right)$$

Answer 3

Simply put, the determinant really is a Jacobian determinant from a transformation.

See, $\sigma$ is outside the exponential because it is a location scale family. If $y\sim N(\mu,\sigma)$ and $x\sim N(0,1)$, then $$ f_y(y) = \dfrac{1}{\sigma}f_x(\dfrac{y-\mu}{\sigma}). $$ This is because the Jacobian of the transformation is $\dfrac{1}{\sigma}$

One way to generalize "location scale" to a multivariate context is through elliptically contoured distributions. If $y\sim N_m(\mu,\Sigma)$ and $x\sim N_m(0,I)$ then $$ f_y(y) = |\Sigma|^{-m/2}f_x((y-\mu)'\Sigma^{-1}(y-\mu)). $$ In the latter case, the Jacobian is a matrix $\Sigma^{-m/2}$, and to finish the transformation we need its determinant.