I've been working on using the Dirichlet clustering algorithm to cluster users based on their behavior.It's an unsupervised task. While the Dirichlet clustering algorithm does a good job of overcoming the problem of predetermining the number of clusters, I still don't see how I can evaluate the quality of the clustering. I also understand that "quality of clustering" is a relative term when it comes to clustering. However, I was wondering if there was a way to test my clustering.I have a large feature set, and I want to pick the best subset of features. As this is an unsupervised problem, I don't have a target variable, so I can't use random forests to rank my features based on their importance.
I came across this link : Feature selection for clustering problems
And the last answer mentioned something interesting :
All you need is a criterion of the clustering quality. Here is the idea: you split the data on train and test, build clustering on train part; use this clustering to cluster each of the element of the test set ( by the closest cluster); build a separate clustering on the test set; find similarity of the clustering in the test with the predicted clustering. This similarity is the criterion of clustering quality. Now, how to measure this similarity is up to you. Once you get it, you select the subset of features to maximize this similarity.
(1) What are the similarity measures that can be used ? Can I use some sort of a hypothesis test (such as KS test) ?
Will the distribution of average posterior probabilities in the training and test set be similar ? Null hypothesis, in this case being that they are similar.
Any help would be appreciated ! Thanks.