12
$\begingroup$

While trying Gaussian Mixture Models here, I found these 4 types of covariances.

'full' (each component has its own general covariance matrix),
'tied' (all components share the same general covariance matrix),
'diag' (each component has its own diagonal covariance matrix),
'spherical' (each component has its own single variance).

I googled a lot to find more details about each of these types but found very high-level descriptions (such as this) only.

Appreciate if someone can help me understand these, or at least direct me to somewhere I can read about these.

$\endgroup$
13
$\begingroup$

A Gaussian distribution is completely determined by its covariance matrix and its mean (a location in space). The covariance matrix of a Gaussian distribution determines the directions and lengths of the axes of its density contours, all of which are ellipsoids.

These four types of mixture models can be illustrated in full generality using the two-dimensional case. In each of these contour plots of the mixture density, two components are located at $(0,0)$ and $(4,5)$ with weights $3/5$ and $2/5$ respectively. The different weights will cause the sets of contours to look slightly different even when the covariance matrices are the same, but the overall shapes of individual contours will still be similar for identical matrices.

Figure

Clicking on the image will display a version at higher resolution.

NB These are plots of the actual mixtures, not of the individual components. Because the components are well separated and of comparable weight, the mixture contours closely resemble the component contours (except at low levels where they may distort and merge, as shown in the center of the "tied" plot for instance).

  • Full means the components may independently adopt any position and shape.

  • Tied means they have the same shape, but the shape may be anything.

  • Diagonal means the contour axes are oriented along the coordinate axes, but otherwise the eccentricities may vary between components.

  • Tied Diagonal is a "tied" situation where the contour axes are oriented along the coordinate axes. (I have added this because initially it was how I misinterpreted "diagonal.")

  • Spherical is a "diagonal" situation with circular contours (spherical in higher dimensions, whence the name).

This exhibit a gamut from the most general possible mixture to a very specific kind of mixture. Other (fussier) restrictions are possible, especially in higher dimensions where the numbers of parameters grow rapidly. (A covariance matrix in $n$ dimensions is described by $n(n+1)/2$ independent parameters.)

$\endgroup$
  • $\begingroup$ Great answer. Thank you. One last question. Are these the only 4 types? or are there any other types too? $\endgroup$ – Bee Feb 3 '18 at 19:35
  • 1
    $\begingroup$ I'm sure you can imagine various intermediate cases where some components share certain properties and other components do not, or where some characteristics of all components are pre-specified. For instance, the most extreme example would be the "spherical" situation where the variance is required to be (say) $1$ in all components. $\endgroup$ – whuber Feb 3 '18 at 19:40
  • $\begingroup$ Thanks. I just compared the quoted description in my post and your answer. In mine, 'Tied' is the only one which is shared by all components. But in yours, 'Full' seems to be the only one which NOT shared by each component. I just feel like these 2 are contradicting. (I'm sure I'm missing something). Do you mind explaining that? Thanks a lot. $\endgroup$ – Bee Feb 3 '18 at 19:47
  • $\begingroup$ I don't see any contradiction: I have faithfully represented precisely the conditions you outline. Indeed, I did not refer to any other sources to create these images. $\endgroup$ – whuber Feb 3 '18 at 19:49
  • 1
    $\begingroup$ Thank you--I see what you mean. I'll update the explanation to reflect that. $\endgroup$ – whuber Feb 3 '18 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.