Significance of a cluster with respect to the whole dataset

Question

I mainly am a machine learning guy and my knowledge of statistics is a bit limited. I have a (small) dataset of objects described by two categorical variables:

$$ \begin{array}{|l|l|l|l|l|} \hline & \mathsf{'l'} & \mathsf{'u'} & \mathsf{'b'} & \mathsf{Tot} \\ \hline \mathsf{'2'} & 15 & 9 & 9 & 33 \\ \hline \mathsf{'3'} & 8 & 5 & 2 & 15 \\ \hline \mathsf{Tot} & 23 & 14 & 11 & 48 \\ \hline \end{array} $$

that obtaining a partition of 6 clusters. One of which is interesting from my point of view:

$$ \begin{array}{|l|l|l|l|l|} \hline & \mathsf{'l'} & \mathsf{'u'} & \mathsf{'b'} & \mathsf{Tot} \\ \hline \mathsf{'2'} & 8 & 3 & 2 & 13 \\ \hline \mathsf{'3'} & 3 & 0 & 0 & 3 \\ \hline \mathsf{Tot} & 11 & 3 & 2 & 16 \\ \hline \end{array} $$

From a perspective of frequencies in the whole dataset the probability of obtaining an object of type ('l', '2') is $p_D=P(r=\mathsf{'2'} \wedge c=\mathsf{'l'} )=\frac{15}{48}=0.3125$, whereas for the cluster is $p_C=P(r=\mathsf{'2'} \wedge c=\mathsf{'l'} )=\frac{8}{16}=0.5$

which is an increase of $\frac{p_C}{p_D}=1.6$.

Now, I have been asked about the significance of this "enrichment" given by the clustering with respect to the whole dataset.

I was thinking about a randomization test in which I build the null distribution assigning randomly the cluster labels and calculating the the ratio $\frac{p_{C'}}{p_D}$ (where $C'$ is the randomized clustering) and testing the probability under the null hypothesis of obtaining a ratio of value at least of $1.6$.

Do you think this is a valid procedure? Can you suggest something better?

Answer 1

To me, this appears to be pretty much random.

At 48 objects, this is not surprising. This is just too little data to yield anything substantial... in particular with this many degrees of freedom.

Testing against a random partitioning will not work. This has been attempted before, but never worked. The null hypothesis is "randomly assigned to clusters"; all you know after rejecting this hypothesis is that your algorithm looked at the data instead of throwing coins.

Don't blindly trust a clustering result. And don't trust an evaluation measure either. Instead: can you explain what the algorithm found? I.e. does the pattern make sense? If not, discard the result. Otherwise, test the pattern on fresh data, and try to use it somehow and check if it is useful.