Suppose in a classification we have a dataset with many features and their class, we want to select some features using which we can construct a classifier.

We perform the cluster evaluation for the given dataset taking different features (or derived features) and find cluster purity and entropy.

Can these purity and entropy calculated for certain feature vector be used to predict the classification performance of the classifier constructed for those set of features.

For eg.

I have set of features vector {A,B,C,D,E,F} and n no dataset. on selection of features there are many possible feature vectors I am considering two: X = {A,B,C,D} and Y = {A,E,F}. Now I perform cluster evaluation and find cluster purity for both X and Y. Can these purity(X) and purity(Y) predict which one is a better selection of feature for classification.

I think this purity may be a lower bound for the performance of classification.

PS: The clustering used is assumed to be k-means clustering with k = no of classes.

  • $\begingroup$ You ought to define/explain purity and entropy in the question - the criterions assessing the quality or success of the clustering partition for you. $\endgroup$ – ttnphns May 20 '16 at 7:47
  • $\begingroup$ Entropy: The degree to which each cluster consists of objects of a single class.Purity: Another measure of the extent to which a cluster contains objects of a single class. Their detailed mathematical formulation could be found in text www-users.cs.umn.edu/~kumar/dmbook/ch8.pdf on page no 63 of the pdf $\endgroup$ – Shaleen Jain May 23 '16 at 9:04
  • $\begingroup$ I have an experimental proof with my data set, but not sure that whether these results holds good for all. $\endgroup$ – Shaleen Jain May 26 '16 at 4:16


Clustering is expensive. If you want to do it right, you need to spend days to improve your preprocessing, select features etc. It's at least as hard as your original problem. It's not as easy as throwing your data into k-means. The results og k-means tend to be quite unpredictable and unreliable. So it's very likely to lead you to wrong conclusions. You always need to double-and-triple-check clustering results!

Even worse: the perfect set of features may reveal an interesting other structure, found by the clustering. Your appriach will consider this to be bad because the clustering found something else - but that is its job, finding new things.

There are reasonably good classifiers that can be trained for about the cost of running k-means just once. So why don't you train a classifier to predict classification performance.

  • $\begingroup$ @anony-mouse I have very small data set, so I don't have any memory and time cost issues, and I already have tried with different features and found the results too, but I just wanted to get proof that any X feature set is best because even a cluster performance evaluation shows to have the highest purity and lowest entropy. PS: Thanks a lot for the input. but can we think in that way, or am I missing something $\endgroup$ – Shaleen Jain May 20 '16 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.