I just start working with Gaussian mixture models and I just confused about some information, which I really would like to make sure that I understand the model very well.

A Gaussian mixture model is a weighted sum of $m$-components Gaussian densities. Hence, each component is assumed to be Gaussian.

My questions are:

1- As I understand the mixture components are used to describe the dependencies structures. So, do these components have any information about the univariate margin of each variable?

In other words, the joint distribution function includes the information of the margin and the dependencies structures. Unlike the copula models, we cannot decouple the margins from the dependencies structures in, for example, Gaussian mixture models. So, the joint distribution includes all the information about margins and the dependency among variables. So, if the mixture components describe the dependencies structures, from where can I get the information of the margins?

By margin, I mean the distribution of each variable.

2- Gaussian mixture models can describe any type of distributions and that may require a very large number of mixture components. Is that correct? or there is no relationship between the type of the mixture dependencies and the number of mixture components? For example, if I have very complex mixture dependencies. From the scatter plot, I find the dependence takes circle shape. Hence, I will need to, say, 20 mixture components to describe this type of dependency.

  • $\begingroup$ @Glen_b amazing help. Its really much more appreciated. $\endgroup$
    – Maryam
    Jul 31, 2018 at 18:17
  • $\begingroup$ The question is unclear: what do you mean by "univariate margin" and "each variable"? The marginal of a multivariate Gaussian mixture is a univariate Gaussian mixture. $\endgroup$
    – Xi'an
    Sep 4, 2018 at 6:00
  • $\begingroup$ @Xi'an Thank you so much for your comment. Suppose I have the Iris dataset. Then, Suppose that I only take two variables, Speal length and Speal width. Then, what are the marginal distributions of Sepal length and Sepal width? Are the marginal distribution of Sepal length is a univariate Gaussian? I know that the marginal distribution of each component is univariate Gaussian, but am asking about the marginal distribution of the random variables. $\endgroup$
    – Maryam
    Sep 4, 2018 at 7:33
  • $\begingroup$ @Glen_b, To make my question much more clear I add a comment to explain my question. I just wonder if you would like to see my comment. Many thanks. $\endgroup$
    – Maryam
    Sep 4, 2018 at 8:22
  • 1
    $\begingroup$ As indicated in the first comment, the marginal of a multivariate Gaussian mixture is a univariate Gaussian mixture. $\endgroup$
    – Xi'an
    Sep 4, 2018 at 10:45

2 Answers 2


(Brief advice/discussion that might not be a full answer but now exceeds a suitable comment)

  1. Mixtures can be used to describe dependence; the components themselves might even have no dependence and yet the dependence could still be modelled (if less parsimoniously). However mixtures are not used only to describe dependence structures; it's something mixtures can be used for, but that's not all they're for.

  2. The margins will be a consequence of the specific mixture. If you model the joint distribution by a mixture it will approximate both the dependence and the margins; once you have a joint distribution, margins can be obtained by integration or may be approximated to a desired accuracy via a simulation.

  3. It may indeed take many components to adequately model a distribution with a mixture. If your ring-structure has a roughly Gaussian cross-section (from center to outside, say) then it might take dozens of components. If it's not close to Gaussian in cross section you may need to increase that number considerably. It may be that much more parsimonious descriptions could be obtained by other choices (such as the product of some circular distribution for the angle with some distribution on the distance of the points from the center, in effect parameterizing the thing in polar coordinates).


Answer to question 1:

The way to test the distribution of each variable can be done through tests like Kolmogorov-Smirnov. The components that are being used for the mixture models, describe as you have said, the "dependency structures" and I really have not heard of anything like taking information about distributions of individual variables through them. A Gaussian model, in addition is a "normal distribution" model which assumes that your components are in normal distribution.

Answer to question 2:

This paper describes an algorithm using copulas in order to achieve correlation clustering with heterogeneous data (continuous binary, categorical). Then this could help you in order to answer your questions about how different distributions can be described. Now, if you want to predict somehow the number of components this paper describes how you can do it. Still, there is a dependency between margins and the model you should use and I do not understand how you can tell if a "mixture dependency" is more complex than another a priori, or else how you can compare two scatter plots. I mean it is easy to compare linear and non-linear associations, but how you can compare two seemingly non-linear, by complexity?

  • $\begingroup$ To compare the complexity of the model, I can use a scatter plot. For example, tail dependencies are less complex than circle dependencies. For tail dependence may need to use less Gaussian components, however, for circle one, we may need to use very large number of Gaussian components. $\endgroup$
    – Maryam
    Jul 30, 2018 at 9:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.