1
$\begingroup$

I've been working on using the Dirichlet clustering algorithm to cluster users based on their behavior.It's an unsupervised task. While the Dirichlet clustering algorithm does a good job of overcoming the problem of predetermining the number of clusters, I still don't see how I can evaluate the quality of the clustering. I also understand that "quality of clustering" is a relative term when it comes to clustering. However, I was wondering if there was a way to test my clustering.I have a large feature set, and I want to pick the best subset of features. As this is an unsupervised problem, I don't have a target variable, so I can't use random forests to rank my features based on their importance.

I came across this link : Feature selection for clustering problems

And the last answer mentioned something interesting :

All you need is a criterion of the clustering quality. Here is the idea: you split the data on train and test, build clustering on train part; use this clustering to cluster each of the element of the test set ( by the closest cluster); build a separate clustering on the test set; find similarity of the clustering in the test with the predicted clustering. This similarity is the criterion of clustering quality. Now, how to measure this similarity is up to you. Once you get it, you select the subset of features to maximize this similarity.

(1) What are the similarity measures that can be used ? Can I use some sort of a hypothesis test (such as KS test) ?

Will the distribution of average posterior probabilities in the training and test set be similar ? Null hypothesis, in this case being that they are similar.

Any help would be appreciated ! Thanks.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ You are mixing up a nonparametric hypothesis, the Kolmogorov-Smirnov test (KS test) with Bayesian methods (posterior probability distributions). What you do need is a distance measure which is a form of clustering criteria. Clustering is unsupervised learning so I do not see how splitting the data into a test and a training set applies. I am also not familiar with Dirichlet clustering. $\endgroup$
    – Michael R. Chernick
    Mar 28, 2017 at 14:25
  • $\begingroup$ Would you suggest something like the Dunn Index ? $\endgroup$
    – red_devil
    Mar 28, 2017 at 18:15
  • $\begingroup$ That answer you refer to is overfit to k-means. A clusterer is not a classifier, most cannot label a test set. It doesn't matter whether you would use Silhouette or any other index, if there is no good way to transfer the clustering to unseen data... $\endgroup$
    – Has QUIT--Anony-Mousse
    Mar 29, 2017 at 20:10
  • $\begingroup$ Also, don't compare across different feature sets. They will not be well comparable if you used different features. $\endgroup$
    – Has QUIT--Anony-Mousse
    Mar 29, 2017 at 20:11

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.