Using R, I ran the K-means algorithm on a dataset with 1m+ rows. Using elbow plot, the optimum no. of clusters was found to be 3. Now each data point is assigned a cluster from the set {1,2,3}. But I'm confused about how to validate the model (apart from the ratio of tot.withinss and betweenss) and is it possible to create a confusion matrix for the same?
3
-
$\begingroup$ Do you have a reference, ground truth partition for the dataset (not necessarily 3 clusters)? $\endgroup$– ttnphnsCommented Mar 4, 2021 at 17:41
-
$\begingroup$ The dataset is labelled, if that's what you're referring to. I'm just finding it hard to grasp the primary objective of clustering. What good are the clusters if we aren't predicting anything but just sorting them into different groups based on similarities? $\endgroup$– Aabhas VijCommented Mar 4, 2021 at 20:44
-
$\begingroup$ I expect Wikipedia should explain where cluster analysis is needed and is worth. And this thread briefly lists approaches to validate clustering results stats.stackexchange.com/q/195456/3277 $\endgroup$– ttnphnsCommented Mar 4, 2021 at 20:51
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
For what do you want a confusion matrix in kmeans? You have unlabeled observations, you have no guarantee that the found clusters are correct even with tot. Furthemore, you are not predicting! You do not predict a cluster/centroid, you are looking at euclidean distances of observations and shift centroids, which are initialized starting positions chosen by random.
Besides the elbow method is a 'non-mathematical' solution as you can go further down the elbow and get even smaller and smaller residual sum of squares. You look at the highest drop in RSS. There is normally no real definite mathematical point, that we have EXACTLY these clusters.
For the part of validating, well in theory you would look at groups which were already defined by parts of your business e.g. 'smart shoppers' where smart shoppers have x attributes wih n manifestations of x. e.g. smart shoppers buy 30 products of category "animal", if you can see your centroid in the middle of points around that amount of bought products, its probably right.
Sometimes we have predefined groups by market research. in germany we would call it 'sinus milieus' which means what attitude people have to politics, life and so on. Thus, sometimes you validate by theory.
In summary: You do not predict centroids, you are just shifting centroids until the euclidean distances to all points, seem 'ok' for the algorithm. There is no sense in using a confusion matrix,--> no probability
Validating: you cant really validate something but look if your groups reflect somehow predefined groups, which you will find through marketing theory, or other theories.
-
$\begingroup$ Your answer makes sense. Paraphrasing my problem statement - Is it possible to make predictions on a labelled dataset by the means of clustering? I have a dataset where the target variable has 3 categories. $\endgroup$ Commented Mar 4, 2021 at 20:37
-
$\begingroup$ @AabhasVij but what for? Yes you can, same as you can do classification by assigning the classes at random, though neither of the approaches gives you any guarantees of optimality. $\endgroup$– TimCommented Mar 4, 2021 at 22:17