I have one dataset of two variables (x,y). When the data is plotted in a 2D diagram, I see some data points create a good cluster, while the other data points are scattered randomly.

Here is an example:

enter image description here

(These data points were collected through an experiment. The weight is equal among all points. After plotting the data points, I found that some points can create a cluster. Then, I changed the color of these points to green color to show those better.)

Based on the plot, I have two clusters: (1) Green points (2) Red Points. Each data point in both clusters has two variables: X, Y.

My question is that:

How can I conduct a statistical test to (statistically) show that the Green cluster has lower entropy than the red cluster?

  • H0: No differences

  • Ha: Green cluster is statistically better (Less Entropy) than the red cluster.

I actually want to show that among all data points (Red + Green), the green data points create a good cluster.

  • $\begingroup$ How were the clusters identified, exactly? $\endgroup$ – whuber Aug 4 '16 at 22:20
  • $\begingroup$ These data points were collected through an experiment. The weight is equal among all points. After plotting the data points, I found that some points can create a cluster. Then, I changed the color of the points with green color to shows those better. I want to discuss that some data points (green data points) can create a cluster. To this end, I need to statistically show what I told in the question. $\endgroup$ – Harry UNL Aug 5 '16 at 3:19
  • $\begingroup$ Because the identification of the cluster was subjective (albeit, no doubt, well informed by expertise) and irreproducible, no statistical test can provide reliable results--and in particular, you cannot sanctify your judgment by applying any such test. Since it is self-evident that the green cluster has much lower entropy than the red points, why not just compute and report the resulting entropies? $\endgroup$ – whuber Aug 5 '16 at 12:15

It's more about the ability to apply an algorithm which can discern the two clusters. Yes, you can see the two clusters, but typical statistical tests of e.g. means won't help since objects colored green are surrounded by red. Thus, try K-nearest neighbors (kNN), with $k=3$. This would essentially involve selecting each object one at a time, finding out the majority class (e.g. color) among the 3 closest neighbors of the object based on Euclidean distance, and then assigning the majority class to the object. (for example, if 2/3 of the 3 closest neighbors are green, then you assign the predicted class of green to the test object). Here, you are predicting class membership based on the majority class (green or red) among the 3 closest neighbors. Simply increment by 1 the elements in a confusion matrix to determine classification accuracy, call this $A$. Since you know the true class (colors shown), you can rerun the same steps but only this time randomly shuffling the true class feature (red/green) over all objects. Repeat this $B=10000$ times, calculate $A^*$ for each iteration, and the number of times the classification accuracy of objects after shuffling (the color feature) $A^*$ exceeds $A$ divided by $B$ is the statistical p-value. This is merely called a randomization test, or empirical p-value testing.

FYI - kNN is the optimal first choice to identify a smaller cluster of objects buried inside of other objects in other clusters. Linear discriminant analysis commonly won't pick this up -- and your dataset is the classic picture for which kNN can do better than many others. Support vector machines (SVM) with a radial basis function (RBF) kernel can pick up the green cluster, but it's computationally more expensive than kNN, and therefore violates Occam's Razor - simpler is better.


Clustering is typically a very subjective exercise. Whether clusters are good or not depend on the problem you are solving.

If you want a numeric way to check for low entropy, you can try mean distance between observations or RMS distance between observations.

Final point is that it's not a good idea to do clustering on 2 variables. We do clustering because there are so many variables that we can't visualise. If we can plot then better to check visually and create.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.