1
$\begingroup$

Using R, I ran the K-means algorithm on a dataset with 1m+ rows. Using elbow plot, the optimum no. of clusters was found to be 3. Now each data point is assigned a cluster from the set {1,2,3}. But I'm confused about how to validate the model (apart from the ratio of tot.withinss and betweenss) and is it possible to create a confusion matrix for the same?

$\endgroup$
3
  • $\begingroup$ Do you have a reference, ground truth partition for the dataset (not necessarily 3 clusters)? $\endgroup$
    – ttnphns
    Mar 4, 2021 at 17:41
  • $\begingroup$ The dataset is labelled, if that's what you're referring to. I'm just finding it hard to grasp the primary objective of clustering. What good are the clusters if we aren't predicting anything but just sorting them into different groups based on similarities? $\endgroup$
    – Aabhas Vij
    Mar 4, 2021 at 20:44
  • $\begingroup$ I expect Wikipedia should explain where cluster analysis is needed and is worth. And this thread briefly lists approaches to validate clustering results stats.stackexchange.com/q/195456/3277 $\endgroup$
    – ttnphns
    Mar 4, 2021 at 20:51

1 Answer 1

1
$\begingroup$

For what do you want a confusion matrix in kmeans? You have unlabeled observations, you have no guarantee that the found clusters are correct even with tot. Furthemore, you are not predicting! You do not predict a cluster/centroid, you are looking at euclidean distances of observations and shift centroids, which are initialized starting positions chosen by random.

Besides the elbow method is a 'non-mathematical' solution as you can go further down the elbow and get even smaller and smaller residual sum of squares. You look at the highest drop in RSS. There is normally no real definite mathematical point, that we have EXACTLY these clusters.

For the part of validating, well in theory you would look at groups which were already defined by parts of your business e.g. 'smart shoppers' where smart shoppers have x attributes wih n manifestations of x. e.g. smart shoppers buy 30 products of category "animal", if you can see your centroid in the middle of points around that amount of bought products, its probably right.

Sometimes we have predefined groups by market research. in germany we would call it 'sinus milieus' which means what attitude people have to politics, life and so on. Thus, sometimes you validate by theory.

In summary: You do not predict centroids, you are just shifting centroids until the euclidean distances to all points, seem 'ok' for the algorithm. There is no sense in using a confusion matrix,--> no probability

Validating: you cant really validate something but look if your groups reflect somehow predefined groups, which you will find through marketing theory, or other theories.

$\endgroup$
2
  • $\begingroup$ Your answer makes sense. Paraphrasing my problem statement - Is it possible to make predictions on a labelled dataset by the means of clustering? I have a dataset where the target variable has 3 categories. $\endgroup$
    – Aabhas Vij
    Mar 4, 2021 at 20:37
  • $\begingroup$ @AabhasVij but what for? Yes you can, same as you can do classification by assigning the classes at random, though neither of the approaches gives you any guarantees of optimality. $\endgroup$
    – Tim
    Mar 4, 2021 at 22:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.