3
$\begingroup$

Does the Gaussian Mixture model require binary and multiclass response/target variable (classification), or can the target vector consist of all real numbers (continuous variable, regression)? Why is it called an unsupervised learning algorithm if it trains on target observations anyway? Or is that not a requirement and it can run without labels?

$\endgroup$
7
  • $\begingroup$ Some RBF-NN’s have GMM’s as their core. $\endgroup$ – EngrStudent Jul 9 '20 at 12:34
  • $\begingroup$ can u explain more $\endgroup$ – develarist Jul 9 '20 at 12:46
  • 1
    $\begingroup$ A gaussian mixture is unsupervised learning. It takes in a set of values and a fixed number N of mixture elements and typically uses expectation maximization to set parameter values for those N Gaussian distributions. The output is the weighted array of distributions, but there are different ways to use those models. You can go from single measurement to probability of membership for each member, or even for the ensemble. In the radial basis function neural network, the internal nodes can be gaussian distributions, so you can use your GMM parameters in the RBF-NN. $\endgroup$ – EngrStudent Jul 9 '20 at 14:33
  • 1
    $\begingroup$ You presume supervised, but it is unsupervised. If you have labels then you know both members and number of components, so the problem is fitting multivariate normal to each class, then adjusting weights for ensemble. That problem is much simpler than what GMM solves. GMM does not get membership so it is unsupervised; it runs without labels. $\endgroup$ – EngrStudent Jul 9 '20 at 14:40
  • 1
    $\begingroup$ The GMM can be used several ways. It can output class membership, and that is recontriving the input space into an output space in useful ways. It can also be inverted and give window of domain(input) associated with that output probability. If you had several probabilities, split up by members, then you might be able to substantially limit the potential input variables, maybe even down to a single point. There is a lot you can do with a GMM. (cse.psu.edu/~rtc12/CSE586Spring2010/lectures/…) $\endgroup$ – EngrStudent Jul 9 '20 at 15:08
2
$\begingroup$

Here is a vignette on Gaussian Mixtures in R:
https://cran.r-project.org/web/packages/sBIC/vignettes/GaussianMixtures.pdf

Here is the example R code from the "Gaussian Mixtures" library for a Gaussian Mixture Model, note in particular the lack of labels and the presence of pre-specified number of cluster components (4) and the specification that all variances are equal "V":

#load the data
library(MASS)
data(galaxies)
X = galaxies / 1000

#this is the gmm library
library(mclust, quietly=TRUE)

#fit to a cluster
fit = Mclust(X, G=4, model="V")

#make summary of fit
summary(fit)

This is the output of the code:

> summary(fit)
--------------------------------------------------- 
Gaussian finite mixture model fitted by EM
algorithm 
--------------------------------------------------- 

Mclust V (univariate, unequal variance) model with
4 components: 

 log-likelihood  n df       BIC      ICL
      -199.2545 82 11 -446.9829 -466.264

Clustering table:
 1  2  3  4 
 7 35 32  8 
> 

The output of this is a model, in particular it is a set of parameters that can go into an analytic (aka symbolic) relationship.

Here is how we get the list of parameters

> fit$parameters

Here is the list of parameters:

$pro
[1] 0.08440664 0.38615546 0.36963554 0.15980236

$mean
        1         2         3         4 
 9.707481 19.804456 22.877670 24.435158 

$variance
$variance$modelName
[1] "V"

$variance$d
[1] 1

$variance$G
[1] 4

$variance$sigmasq
[1]  0.1772940  0.4353339  1.2526260 34.1224385

$variance$scale
[1]  0.1772940  0.4353339  1.2526260 34.1224385

It, as in the parameter list, can be applied in a number of ways. It can be applied forward, backward, and several versions of sideways. Think of it as a collection of bubble-surfaces in a space. (bubble surface = surface of constant probability density)

Some things you could look at:

  • in vs. out for any single bubble
  • in vs. out for the collection of bubbles
  • distance from bubble center for any single bubble
  • distance from aggregated center for any permutation of bubbles
  • are there too many or too few bubbles - how to get N correct
  • is the space really a membrane, can you use PCA to reduce your bubble dimensionality
  • do you have outlier bubbles where most are connected and some aren't
  • are your bubble diameters close enough to equal to consider them equal (kmeans)
  • how to the parameters vary when you repeat the process (is your fit robust)
  • do you have pathological points
  • do you have pathological bubbles
  • how much of the volume of the convex hull of the space is shared by 2 or more bubbles
  • what does the set of planes of equal probability of cluster membership look like (classification boundaries)

Update:

Consider this image, and imagine it was drawn better. Each family of 10 or so dots is really its own Gaussian distribution. It has its own mean and covariance matrix.

enter image description here

Together these 14 individual distributions in a 2d space can make a single Gaussian Mixture model. You weight each individual component by which fraction of total points it contains, and you would get about 1/14 for your average weight.

If you had a bigger space, and more time, you could make a 2d space with any number of individual Gaussian distributions, each having their own mean and covariance. You could then create a single Gaussian Mixture from all of that arbitrary number.

Check the image here out:
https://towardsdatascience.com/mixture-modelling-from-scratch-in-r-5ab7bfc83eef

$\endgroup$
3
  • $\begingroup$ A separate question about GMM: in the sklearn example in the following link, someone starts with a dataset with two columns only, but then tries to vary the number of mixture components from 1 to 6. I don't understand how can there be 6 mixture components to try, when the original data only has 2? Aren't the number of columns in the data equal and cannot be greater than the number of mixture components, vice versa? scikit-learn.org/stable/auto_examples/mixture/… $\endgroup$ – develarist Jul 10 '20 at 15:35
  • $\begingroup$ Think about a 2D Cartesian coordinate system. It has two axes: X, and Y. You can take a pencil and draw 1000 completely separate clusters of dots. If you draw them well, each cluster of dots could be its owner component in a mixture model. Overall then, there would be 1000 elements in the mixture, each of which was a Gaussian distribution centered on a single cluster of dots. $\endgroup$ – EngrStudent Jul 10 '20 at 17:06
  • $\begingroup$ i try to follow your analogy but can't quite get it. the data starts with only 2 columns/features. if these are 2 distinct mixture components, then I would expect 2 clusters to be found by GMM. If I intentionally boost up or target the number of clusters to be more than 2, is the original data being altered or split up somehow for the component (and thus cluster) count to suddenly go from 2 to 6? $\endgroup$ – develarist Jul 13 '20 at 9:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.