Choosing a model for my unsupervised machine learning problem

Question

I need to choose a model for unsupervised machine learning problem. There are 4 clusters in 3D space. These are my requirements:

• I will run the same model multiple times with different training data (it is for real-time application).
• Size of training data is expected to be around 400 points.
• I can assume that points for each of the clusters are drawn from a Gaussian distribution. This is not necessary requirement to be present in the model.
• I need to get 4 points that represent "centers" of clusters.
• In prediction time, for each new point I need some kind of number for each cluster that will represent probablity of belonging to the cluster.
• I will have a lot of outliers, assume around 30%.

I have tried Gaussian mixture model, and it works very good when I don't have outliers. Unfortunately, this model is very sensitive to outliers.

Any suggestions how to handle the outliers with Gaussian mixture model? Or should I go with completely different model?

Answer 1

Here are a couple suggestions, given that Gaussian mixture models work well for you in the absence of outliers.

To increase robustness to outliers, you could use a trimmed estimator for Gaussian mixture models instead of fitting with the standard EM algorithm. Some relevant papers:

• Neykov et al. (2007). Robust fitting of mixtures using the trimmed likelihood estimator.
• Gallegos and Ritter (2009). Trimmed ML Estimation of Contaminated Mixtures.

Instead of Gaussian mixture models, you could also consider student T mixture models. This will give the same properties you want (e.g. ability to compute cluster centroids and membership probabilities). Student T distributions have heavier tails than Gaussians, which increases robustness to outliers. Some relevant papers:

• Peel and McLachlan (2000). Robust mixture modelling using the t distribution.
• Svensen and Bishop (2005). Robust Bayesian Mixture Modelling.
• Archambeau and Verleysen (2007). Robust Bayesian clustering.

Answer 2

Look, this might not be the best idea, but I'll suggest it anyway.

You say that you are dealing with an unsupervised ML problem, but said that:

• you can assume that in this 4 dimensional space, your data is a multivariate Gaussian;

• your model works nicely when there aren't many outliers

If you can afford not assigning a class to an outlier observation, go with DBSCAN as mentioned, or HDBSCAN, where you can even work without the Gaussian assumption.

Now, my idea is: based on the fact that your problem works well when there aren't outliers I suggest you turn this problem into a supervised one provided that:

• you have a consistent number clusters every time you run the clustering algorithm

• the points are consistently assigned to the same clusters

When you have a labeled data set (after clustering), you can use the most appropriate classification algorithm to assign new points to those clusters, or work based on the predicted probabilities (hint, KNN works wonders in low dimensional spaces, such as yours).

If everything is working, outliers should be classified with a low probability of belonging to any clusters, and you can program your online application to handle these cases.

Answer 3

I am assuming that the required four clusters will be of somewhat similar density. If that is the case then you can use a density based clustering approach, DBSCAN has worked well for me. You can find all the clusters that you can during training time and have a threshold on the cluster size to preclude the outliers from being part of your desired clusters.

Using this, and some knowledge about what constitutes as an outlier, you can look into the clusters and see if the outliers (as ~30% seems a bit on the higher end) themselves form a cluster.

Scikit-learn in python has an in-built function for this. sklearn.cluster.DBSCAN

Answer 4

Red flags go off in my head when someone says (paraphrasing heavily)

I can assume property X for my algorithm, but this peroperty X need not be present in the data.

That immediately points to possible performance issues and/ or grounds to revaluate model selection step.

Having said that, I haven't worked with GMM for clustering beyond the base EM algorithm to find density parameters of points. But to my best understanding, clustering using GMM is fuzzy, i.e. depending on implementation methods you can vary cluster assignment based on some criteria of interest.

Working off the above approach I recommend self-organizing maps that semantically group together similar items. As a second step, clustering can be applied on top of it. (Reference)

Also, when working with mixed models, exploring kernel methods can be quite fruitful as now, your decision boundaries can be more expressive in higher dimensional space. Here's a paper that talks theoretically about how to make common clustering methods more robust using kernel, it covers data with mixture of Gaussian densities as well.

Answer 5

Only 3 dimensions and 30% of your data are outliers? That does not seem to fit with what I normally think of as an outlier. Perhaps you can simply transform your variables to log scale or "cap" the outliers so that they are no more than 3 or 4 standard deviations from the mean. This may create clusters of points with no variance, though.