7
$\begingroup$

Displayed below are the contours and their respective covariance matrices according to Andrew Ng's notes (pdf). Why are the first and second contours elliptical and not circular? The variance along both axes is the same.

Here's one last set of examples generated by varying $\Sigma$:

Andrew Ng Gaussian Notes Pg4, three sets of elliptical contours

The plots above used, respectively,
$$ \Sigma = \begin{bmatrix} 1&-0.5\\-0.5 &1 \end{bmatrix}; \qquad \Sigma = \begin{bmatrix} 1&-0.8\\-0.8 &1 \end{bmatrix}; \qquad \Sigma = \begin{bmatrix} 3&0.8\\0.8 &1 \end{bmatrix}. $$

$\endgroup$
8
  • $\begingroup$ Looks like a scaling issue to me. The range is the same, but the length of the plot region are not. $\endgroup$
    – dimitriy
    Feb 1 '18 at 20:17
  • $\begingroup$ So, is what I think right and the contours of the first and third covariance matrices should be circles? $\endgroup$
    – skyquake
    Feb 1 '18 at 20:20
  • $\begingroup$ The only reason they can be ellipses is if the variances are different. You could verify my claim by printing the page and measuring with a ruler. $\endgroup$
    – dimitriy
    Feb 1 '18 at 20:21
  • 5
    $\begingroup$ That the variances are the same is revealed by comparing the widths and heights of the ellipses. This has nothing to do with the eccentricities, which also depend on the correlations. @Dimitriy Scaling is not the explanation. At a correct aspect ratio all three plots would be square, but all three sets of ellipses would still be non-circular. $\endgroup$
    – whuber
    Feb 1 '18 at 20:27
  • 4
    $\begingroup$ @whuber is right. Correlation will also make them ellipses, even if variance is the same. $\endgroup$
    – dimitriy
    Feb 1 '18 at 20:35
9
$\begingroup$

You can understand the shape of the ellipsoid better if you look at the spectral/eigen decomposition of the precision matrix (inverse of the covariance matrix). You want to look at the eigenvalues of this inverse, not the diagonal elements.

Just a supplement to the other answers: for a multivariate Normal with dimension $k$, you can see why algebraically if you follow this. Set the density equal to some level $l$, then: \begin{align*} (2\pi)^{-k/2} |\Sigma|^{-1/2} \exp\left(-\frac{1}{2}(x-\mu)'\Sigma^{-1}(x-\mu) \right) &= l\\ \iff \exp\left(-\frac{1}{2}(x-\mu)'\Sigma^{-1}(x-\mu) \right) &= l'\\ \iff (x-\mu)'\Sigma^{-1}(x-\mu) &= l''.\tag{*} \end{align*} (*) is the formula for an ellipsoid centered at $\mu$. The

For your first covariance matrix, the spectral decomposition of its inverse is $\Sigma^{-1} = P\Lambda P'$, where $$P = \left[\begin{array}{cc} P_1 & P_2 \end{array}\right] = \left[\begin{array}{cc} .707 & -.707\\ .707 & .707 \end{array}\right] $$ and $$ \Lambda = \left[\begin{array}{cc} \lambda_1 & 0 \\ 0 & \lambda_2 \end{array}\right] = \left[\begin{array}{cc} 2 & 0 \\ 0 & 2/3 \end{array}\right]. $$ The reason why it looks "squished" is because the diagonals of $\Lambda$ are not the same. This is because the semi-axes are $P_1/\lambda_1$ (the up and to the right vector) and $P_2/\lambda_2$ (up and to the left). Because $\lambda_1$ is bigger, that means $P_1/\lambda_1$ is a shorter vector.

What if we're used to looking at the covariance matrix, instead of its inverse? Well their spectral decompositions are pretty related. Because $\Sigma^{-1} = P\Lambda P'$ and because $P$ is orthogonal, we have $$ \Sigma = P \Lambda^{-1}P'. $$ Just try multiplying these two decompositions together, and you should get the identity matrix. What this tells us is that these two matrices have the same eigenvectors (and so they have the same principal axes), and the eigenvalues are reciprocals. However, I started off with the precision matrix because that's what is in the formula for the density.

More examples:

If the elements of $x$ are independent, then $\Sigma$ is diagonal, then $\Sigma^{-1}$ is diagonal, then (*) is $$ \frac{(x_1 - \mu_1)^2}{\sigma_1^2} + \frac{(x_2 - \mu_2)^2}{\sigma_2^2} = l''\tag{**} $$ which is still an ellipse, but it's not tilted/rotated.

If the elements of $x$ are independent and moreover they are identical, then $\sigma_1 = \sigma_2$ and (**) turns into a circle.

$\endgroup$
3
  • $\begingroup$ The equation of an ellipse on wikipedia has a $1$ on the RHS, so in this case $$ (x-\mu)^\top \Sigma^{-1}(x - \mu) = 1 $$ How does having $l''$ rather than $1$ change the ellipsoid? Surely it has some impact. $\endgroup$ Mar 12 '21 at 9:33
  • 1
    $\begingroup$ @Euler_Salter what happens if you scale both sides of (*) by $1/l''$? $\endgroup$
    – Taylor
    Mar 28 '21 at 1:48
  • 1
    $\begingroup$ @Euler_Salter I'm late to the party, but it should basically scale the ellipse by $\sqrt{l''}$ equally in each dimension. $\endgroup$
    – Chris
    May 6 '21 at 12:19
6
$\begingroup$

Assume you are visualizing the distribution of a vector called $(X,Y)$ (assumed to have a bivariate normal distribution).

When $X$ and $Y$ have the same variance, the projections of the ellipse on both axes have the same length. This does mean it's a circle. It can be oblique. It's not a circle when $X$ and $Y$ are not independent.

When $X$ and $Y$ are independent, the major and minor axes of the ellipse are aligned with the axes. This does not mean it's a circle either, it can be flattened.

A circle requires both:

  • independence of $X$ and $Y$
  • $X$ and $Y$ having the same variance

This is when the covariance matrix $\Sigma$ is diagonal with a constant diagonal.

$\endgroup$
3
  • $\begingroup$ (+1) But note that your assertions implicitly suppose $(X,Y)$ has a bivariate Gaussian distribution. Otherwise, you should replace "independent" by "uncorrelated." $\endgroup$
    – whuber
    Feb 1 '18 at 21:15
  • 1
    $\begingroup$ Title of the question: "...Multivariate Gaussian...". But I'll add it in my answer because I also felt a doubt when writing it. $\endgroup$ Feb 1 '18 at 21:16
  • $\begingroup$ Understood: but I had taken your introductory sentence referring to "a distribution" as a (legitimate) attempt to generalize the result. $\endgroup$
    – whuber
    Feb 1 '18 at 21:18
0
$\begingroup$

enter image description here

Consider this figure. Notice how both the circle and the dashed diagonal are inside the square. So, the circle is how the contours of the multivariate Gaussian looks when correlation is zero. The dashed diagonal is the contour of the perfectly correlated variables. The ovals (ellipses) are in between, when correlation is not equal zero or one. The length of the square sides represents the variance (standard deviation) of the variables (marginals).

Here, I resized your picture to make the x- and y-axis scales equal, and you can see how the oval fits into a square. I think that the fact that Andrew Ng's plot was not scaled equally just added to the confusion. You can fit all kinds of ovals into the same square. You can have all kinds of contours for the same variances of variables depending on the correlation between them.

enter image description here

The image is from this web site, which has nothing to do with a question asked :)

$\endgroup$
1
  • 1
    $\begingroup$ It would be nice if you could clarify that we need zero correlation AND equal variances for circular contours. $\endgroup$ Feb 1 '18 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.