2
$\begingroup$

I need to find/fit a model for the color of an object. Suppose its color is generally yellow and we have 10000-by-3 data which are pixel values for R, G, B channels.

enter image description here

Firstly I choose a Multivariate Gaussian Model (MVG) and here is output of the learning process:

$$\begin{array}{*{20}{c}} {{\bf{\mu }}_{}^{{\rm{MVG}}} = \left[ {{\rm{150}}{\rm{.4999 , 145}}{\rm{.5921 , 58}}{\rm{.3353}}} \right]}&{{\bf{\Sigma }}_{}^{{\rm{MVG}}} = } \end{array}\left[ \begin{array}{l} {\rm{157}}{\rm{.6437 , 118}}{\rm{.2281 ,- 182}}{\rm{.2751}}\\ {\rm{ 118}}{\rm{.2281 , 121}}{\rm{.3255 , - 158}}{\rm{.3915}}\\ {\rm{ - 182}}{\rm{.2751 ,- 158}}{\rm{.3915, 295}}{\rm{.6557}} \end{array} \right]$$ Also I try to model the color by using a Gaussian Mixture (GMM) with 2 commponents. Here is the output of E-M algorithm:

$$\begin{array}{l} \begin{array}{*{20}{c}} {{\bf{\mu }}_1^{{\rm{GMM}}} = \left[ {{\rm{168}}{\rm{.4785 , 168}}{\rm{.0000 , 20}}{\rm{.5215}}} \right]}&{{\bf{\Sigma }}_1^{{\rm{GMM}}} = } \end{array}\left[ \begin{array}{l} {\rm{ 0}}{\rm{.7710 , 0 , - 0}}{\rm{.7710}}\\ {\rm{ 0 , 0 , 0}}\\ {\rm{ - 0}}{\rm{.7710 , 0 , 0}}{\rm{.7710}} \end{array} \right]\\ \begin{array}{*{20}{c}} {{\bf{\mu }}_2^{{\rm{GMM}}} = \left[ {{\rm{150}}{\rm{.4998 , 145}}{\rm{.5920 , 58}}{\rm{.3354}}} \right]}&{{\bf{\Sigma }}_2^{{\rm{GMM}}} = } \end{array}\left[ \begin{array}{l} {\rm{ 157}}{\rm{.6090 , 118}}{\rm{.2017 ,- 182}}{\rm{.2342}}\\ {\rm{ 118}}{\rm{.2017 , 121}}{\rm{.2982 , - 158}}{\rm{.3553}}\\ {\rm{ - 182}}{\rm{.2342 ,- 158}}{\rm{.3553 ,295}}{\rm{.5886}} \end{array} \right] \end{array}$$

As you can see the value for 2nd component in GMM is the same as the values for MVG.

How we can interpret the values for the first component of GMM? What thing does it represent? What this specific structure of ${{\bf{\Sigma }}_1^{{\rm{GMM}}}}$tells us about the underlying model?

$\endgroup$
5
$\begingroup$

The zero values in the first component's covariance matrix suggest that these are some points (maybe even just two) that have exactly the same value (which must be 168) on the second variable.

The similarity of the parameter estimates of the second component with the overall MVG suggests that this second component contains all other, i.e., almost all observations.

Note that a zero row in the covariance matrix means that it is not invertible. The likelihood for this solution will be degenerate. This suggests that this is a so-called spurious solution (in which a component just fits one or more points that can be fitted perfectly with a degenerate or near-degenerate covariance matrix), and that the EM-algorithm got stuck there. This is a well known issue with Gaussian mixtures. These solutions are usually meaningless (except telling you that there are at least two observations with identical values on the second variable). Chances are that with a different initialisation of your EM-algorithm (you didn't tell us how exactly you ran this) it will give you something more useful.

$\endgroup$
5
  • $\begingroup$ Thank you for the answer. Colud you provide an example of a degenerate as well as near-degenerate covariance matrix? Another question: You says "The likelihood for this solution will be degenerate." So, I interpret it as two distributions shrink too fast to a degenerate/atomic/individual distribution. Is my understanding correct? $\endgroup$ – sci9 Feb 8 '20 at 16:05
  • $\begingroup$ Another question yet! How we can avoid the EM-algorithm from spurious solution? Just run it agian and see the result? $\endgroup$ – sci9 Feb 8 '20 at 16:09
  • 1
    $\begingroup$ Your component 1 covariance matrix is degenerate. It would be "near-degenerate" if you had $10^{-8}$ instead of zero entries. To be honest, I don't understand your "two distributions shrink too fast" interpretation. What happens is that if the EM-algorithm encounters a degenerate covariance matrix, it cannot continue, because the likelihood is already infinite (meaning degenerate) and cannot be improved anymore. Surely the bigger mixture component has not been "shrinking". $\endgroup$ – Lewian Feb 8 '20 at 16:11
  • 1
    $\begingroup$ Second question: The result of the EM-algorithm depends on the initialisation. I have no idea how you did that, but in all likelihood a different initialisation will give you a different solution, which may well be better/non-degenerate. In fact you could use available software such as the mclust R-package of which the authors have put some thought and experience into efficient initialisation. Even there degeneration can happen. One of mclust's ways for dealing with this is to fit a model with suitably constrained covariance matrices. $\endgroup$ – Lewian Feb 8 '20 at 16:14
  • $\begingroup$ Merci for clarifying things up! +1+1+1 $\endgroup$ – sci9 Feb 8 '20 at 16:49
2
$\begingroup$

In the case of the Multivariate Gaussian Model, you are computing the mean and the covariance of a single cluster (so you would expect your points to look like the points in the first answer here), while when you are using a GMM, you try to find two clusters that capture your data the best (see some examples, but for multiple clusters: here or here).

Because in your case one GMM cluster has (almost) the same parameters as the MVG, I expect the other GMM cluster to only contain a few points representing noise (the mean is relatively similar to the mean of the other cluster, but the values of the covariance matrix are small, so these few points are probably super close).

Because your points are 3D, you could try plotting them using a different color for each of the two GMM clusters and see if this hypothesis is indeed true.

$\endgroup$
2
  • $\begingroup$ Thank you for the answer. I've edited the post with a 3D as well as 2D plots. Now what we can say about the data and its estimated model parameters? $\endgroup$ – sci9 Feb 8 '20 at 15:24
  • $\begingroup$ I tend to think Andreea M is right. Another way of checking this is to look at the estimated component proportions. I'd expect the one for the second mixture component (which is almost equal to the overall MVG) to be very high, probably even clearly above 0.9. $\endgroup$ – Lewian Feb 8 '20 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.