I'm trying to make a simple outlier detection program that is able to correctly, or almost correctly, identify values in a data set that could be potential outliers because they don't fall in the distribution of the rest of the values in the same data set.
I can't use supervised techniques like classification or regression because I am not granted any historical, labeled data to train a model with, so I will be using unsupervised techniques, like clustering.
I was going to use k-means clustering, but I read multiple articles saying that k-means works horribly with outliers, and some of them recommended me to try a gaussian mixture model.
I know Gaussian Mixture Models work by creating different clusters that represent different distributions. I am using Spark's (Apache) version of Gaussian Mixture Model and this gives me two columns relevant to my problem: a prediction column that gives me the cluster for which a data point in the data set has been assigned to, and a probability column, which is a column that gives me the probabilities that each value has to be assigned to each one of the clusters. Working with this approach, how can I determine outliers?
I thought of labeling as outliers those values which are on the lower cluster (cluster with the smallest number of points) but this is not a good approach because on the scenario that there are no outliers, there will always be one cluster smaller than the rest since GMM doesn't evenly distributes the values in the clusters. Any alternative approach I could use?