6
$\begingroup$

I am not well versed in statistics. I wanted to know why we use the determinant of the covariance matrix instead of having the covariance matrix itself when writing down the multivariate normal distribution. Why do we do this and what is the intuition behind it?

I have noticed that for a basic Maximum Lkikelihood estimation with normal errors can be written as follows for the univariate case:

$\frac{1}{\sqrt{2\pi \sigma^2}}\cdot \exp(-\frac{1}{2}(\epsilon/\sigma)^2)$

and as follows for the multivariate case:

$\frac{1}{\sqrt{2\pi |\Sigma|}}\cdot \exp(-\frac{1}{2}(\frac{\epsilon}{|\Sigma|})^2)$

What I want to know is why do we use the determinant of the covariance matrix in the multivariate case. In the univariate case we have the variance sigma^2, but in the multivariate case we write the determinant of the variance.

$\endgroup$
  • 2
    $\begingroup$ I don't understand what you are asking about the determinant. The multivariate normal distribution is written with the covariance matrix en.wikipedia.org/wiki/Multivariate_normal_distribution $\endgroup$ – rocinante Mar 13 '14 at 23:50
  • $\begingroup$ I was particularly looking into an MLE function with normal errors. I notice that we dont write the covariance matrix but its determinant and wanted to know why do we need to do this. It must be that the determinant of the variance is all we need and not the whole matrix. I want to know why is this so. $\endgroup$ – Sophie Mar 14 '14 at 0:05
  • 1
    $\begingroup$ Okay, can you actually write the MLE function you're having trouble with in your question, please? We can't help you if you're vague with your question. $\endgroup$ – rocinante Mar 14 '14 at 0:13
  • $\begingroup$ I edited it but I do not know how to write the function in a neater way. Hope this helps $\endgroup$ – Sophie Mar 14 '14 at 0:20
  • 1
    $\begingroup$ I tidied up your mathematics, but as you wrote it, it looks like it's wrong. Where did that come from? $\endgroup$ – Glen_b Mar 14 '14 at 1:16
8
$\begingroup$

Instead of jumping to the multivariate case in matrix form, look at the bivariate case first: enter image description here

Can you recognize the portion of the denominator that is the determinant of the variance-covariance matrix below?

enter image description here

In the univariate case you don't have a determinant because $\sum$ consists of just one term. You don't have another variable, so you don't need to take into account any interaction between them.

$\endgroup$
10
$\begingroup$

Here another practical way to feel more confident about the vector and matrix notation of the multivariate normal. How did the transformation to the multivariate case work and generate $\Sigma_\mathcal{E}$ and $\mathbf{(y-\mu)}'(\Sigma_\mathcal{E}^{-1})(\mathbf{y-\mu})$ in the new joint density? We will show this briefly.

Lets say we have a random vector $\mathbf{y}$ with three variables as in

$$\mathbf{y}=\begin{pmatrix} y_1 \\ y_2 \\ y_3\end{pmatrix}, \mathbb{E}[\mathbf{y}]=\begin{pmatrix} \mu_1 \\ \mu_2 \\ \mu_3\end{pmatrix}, \Sigma_\mathcal{E}= \begin{pmatrix} \sigma_1^2 & 0 & 0 \\ 0 & \sigma_2^2 & 0 \\ 0 & 0 & \sigma_3^2 \end{pmatrix}$$

Clearly when these random values appear we observe them simultaneously and therefore the joint probability of observing particular values of this random vector simultaneously is equal to

$$P(y_1)\cdot P(y_2)\cdot P(y_3)$$

Where $P(...)$ represents the (marginal) probability distribution of the random variables. Since we reason from mutually independent standard normal random variables, we implicitly assume independence between the variables. This is crucial because only then can we simply multiply the marginal probabilities to arrive at joint probabilities. The logic is much the same as throwing two $6$ dots with two dices at once, under natural independence the probability is $(1/6)(1/6)≈2.8\%$. For our values of $y_1,y_2$ and $y_3$ we can apply the univariate Normal PDF, that is

$$P(y_1)=\dfrac{1}{\sqrt{2\pi}\sigma_1}\exp\left(-\dfrac{(y_1-\mu_1)^2}{2\sigma_1^2}\right) $$ $$P(y_2)=\dfrac{1}{\sqrt{2\pi}\sigma_2}\exp\left(-\dfrac{(y_2-\mu_2)^2}{2\sigma_2^2}\right) $$ $$P(y_3)=\dfrac{1}{\sqrt{2\pi}\sigma_3}\exp\left(-\dfrac{(y_3-\mu_3)^2}{2\sigma_3^2}\right) $$

And hence if we multiply we get

$$P(y_1)\cdot P(y_2)\cdot P(y_3)=\left(\dfrac{1}{\sqrt{2\pi}\sigma_1}\right)\left(\dfrac{1}{\sqrt{2\pi}\sigma_2}\right)\left(\dfrac{1}{\sqrt{2\pi}\sigma_3}\right)\exp\left(-\dfrac{(y_1-\mu_1)^2}{2\sigma_1^2}-\dfrac{(y_2-\mu_2)^2}{2\sigma_2^2}-\dfrac{(y_3-\mu_3)^2}{2\sigma_3^2}\right)$$

$$=\left(\dfrac{1}{\sqrt{2\pi}^3\sigma_1\sigma_2 \sigma_3}\right)\exp\left(-\dfrac{1}{2}\left(\dfrac{(y_1-\mu_1)^2}{\sigma_1^2}+\dfrac{(y_2-\mu_2)^2}{\sigma_2^2}+\dfrac{(y_3-\mu_3)^2}{\sigma_3^2}\right)\right)$$

Now notice that

$$\sigma_1^2 \sigma_2^2 \sigma_3^2=\det(\Sigma_\mathcal{E})=\sigma_1^2\left| {\begin{array}{cc} \sigma_2^2 & 0 \\ 0 & \sigma_3^2 \end{array} } \right|=\sigma_1^2 \sigma_2^2 \sigma_3^2 $$

and notice that

$$\mathbf{(y-\mu)}'(\Sigma_\mathcal{E}^{-1})(\mathbf{y-\mu}) = \\= \begin{pmatrix} y_1-\mu_1 & y_2-\mu_2 & y_3-\mu_3 \\\end{pmatrix} \begin{pmatrix} \sigma_1^2 & 0 & 0 \\ 0 & \sigma_2^2 & 0 \\ 0 & 0 & \sigma_3^2 \end{pmatrix}^{-1}\begin{pmatrix} y_1-\mu_1 \\ y_2-\mu_2 \\ y_3-\mu_3 \\\end{pmatrix} $$

$$=\begin{pmatrix} y_1-\mu_1 & y_2-\mu_2 & y_3-\mu_3 \\\end{pmatrix} \begin{pmatrix} 1/\sigma_1^2 & 0 & 0 \\ 0 & 1/\sigma_2^2 & 0 \\ 0 & 0 & 1/\sigma_3^2 \end{pmatrix}\begin{pmatrix} y_1-\mu_1 \\ y_2-\mu_2 \\ y_3-\mu_3 \\\end{pmatrix} $$

$$ = \begin{pmatrix} (y_1-\mu_1)/\sigma_1^2 & (y_2-\mu_2)/\sigma_2^2 & (y_3-\mu_3)/\sigma_3^2 \\\end{pmatrix} \begin{pmatrix} y_1-\mu_1 \\ y_2-\mu_2 \\ y_3-\mu_3 \\\end{pmatrix}$$

$$= \left(\dfrac{(y_1-\mu_1)^2}{\sigma_1^2}+\dfrac{(y_2-\mu_2)^2}{\sigma_2^2}+\dfrac{(y_3-\mu_3)^2}{\sigma_3^2}\right)$$

Therefore we can write the multivariate normal (joint distribution) as

$$\dfrac{1}{\sqrt{(2\pi)^k\det(\Sigma_\mathcal{E})}}\exp\left(-\dfrac{\mathbf{(y-\mu)}'(\Sigma_\mathcal{E}^{-1})(\mathbf{y-\mu})}{2}\right)$$

$\endgroup$
  • $\begingroup$ Was this taken from a book/lecture notes? If so it would be good for it to be attributed properly. $\endgroup$ – Silverfish Dec 17 '14 at 22:43
  • 3
    $\begingroup$ No, it was an example I came up with myself to clarify things. $\endgroup$ – Marc Dec 17 '14 at 23:36
  • 1
    $\begingroup$ You can actually type-set in Latex in your answers using the $ symbol - that's easier than copying over an image of the type-set equations, and makes it more straightforward to edit too! Also the text would be searchable which is often useful. $\endgroup$ – Silverfish Dec 17 '14 at 23:44
  • $\begingroup$ Thanks, I used the Latex format. I didn't manage to make an argument that adjusts the size of the equation so that it fits on one line... Any ideas? $\endgroup$ – Marc Dec 18 '14 at 11:50
  • $\begingroup$ I was going to point you to our help page on these matters, but I realised we don't have much of one! meta.stats.stackexchange.com/questions/1604/… is worth a read. You are likely to get a decent response if you ask a question on Meta ... another thing I do quite often, if I see someone whose MathJax has rendered nicely, is click the "edit" button under their post, copy their code, then readapt it for my purposes. $\endgroup$ – Silverfish Dec 18 '14 at 13:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.