When we do classification and regression, we usually set testing and training sets to help us build and improve models.
However, when we do clustering do we also need to set testing and training sets? Why?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ Yes - for similar reasons as classification/regression. You want to make sure that whatever model you create (say your elbow plot indicates that k=3 in a k-means clustering) is still appropriate to unseen data. $\endgroup$– ilanmanCommented Mar 21, 2017 at 17:50
-
$\begingroup$ Thank you ilanman ;) Also, do you have any recommendations about how to determine the actual number of clusters when we do clusterings such as kmeans? $\endgroup$– rz.HeCommented Mar 21, 2017 at 18:19
-
1$\begingroup$ No. You do not use training and testing in unsupervised learning. There is no objective function in unsupervised learning to test the performance of the algorithm. $\endgroup$– S_DhungelCommented Jul 19, 2017 at 17:04
-
$\begingroup$ We do use something like "training" and "test" data in cluster cross-validation gimmicks. See a brief list of clustering validation approaches. $\endgroup$– ttnphnsCommented May 19, 2023 at 14:26
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
6
Yes, because clustering may also suffer from over-fitting problem. For example, increasing number of clusters will always "increase the performance".
Here is one demo using K-Means clustering:
The objective function of K-means is
$$ J=\sum_{i=1}^{k}\sum_{j=1}^{n}\|x_i^{(j)}-c_j\|^2 $$
With such objective, the lower $J$ means "better" model.
Suppose we have following data (iris data), choosing number of cluster as $4$ will always "better" than choosing number of cluster as $3$. Then choosing $5$ clusters will be better than $4$ clusters. We can continue on this track and end up with $J=0$ cost: just make number of the cluster equal to number of the data points and place all the cluster center on the corresponding the points.
d=iris[,c(3,4)]
res4=kmeans(d, 4,nstart=20)
res3=kmeans(d, 3,nstart=20)
par(mfrow=c(1,2))
plot(d,col=factor(res4$cluster),
main=paste("4 clusters J=",round(res4$tot.withinss,4)))
plot(d,col=factor(res3$cluster),
main=paste("3 clusters J=",round(res3$tot.withinss,4)))
If we have hold off data for testing, it will prevent us to over-fit. The same example, suppose we are choosing large number clusters and put every cluster center to the training data points. The testing error will be large, because testing data points will not overlap with the training data.
answered Mar 21, 2017 at 17:55
-
$\begingroup$ Hi hxd1011, thanks for your quick reply. Another question, do you have any recommendations about how to determine the actual number of clusters when we do clusterings such as kmeans? $\endgroup$– rz.HeCommented Mar 21, 2017 at 18:19
-
$\begingroup$ @rz.He yes, check this answer stats.stackexchange.com/questions/261537/… $\endgroup$ Commented Mar 21, 2017 at 18:23
-
2$\begingroup$ +1 because it is a constructive answer but to play devil's advocate you do know they are 3 clusters. If someone showed this data without any context a 2-cluster solution would work beautifully too. Maybe you even have some of the upper right-most points as outliers to play "real-data-have-outliers" too. It would be much more constructive (and stringent) to look at the coherence between bootstrapped/jittered/subsetted clustering runs using some statistic (eg. cophenetic correlation, Adjusted Rand-Index, etc.). $\endgroup$ Commented Mar 21, 2017 at 21:28
-
1$\begingroup$ And if you don't use k-means? Say, average linkage clustering? I fear thar your answer is overfitting to k-means. $\endgroup$ Commented Mar 22, 2017 at 7:10
-
$\begingroup$ @Anony-Mousse: The answer is particular to k-means as an example but it would be qualitatively the same if DBSCAN or spectral clustering or whatever else it was used. It just shows that a particular metric can be over-fitted. $\endgroup$ Commented Mar 22, 2017 at 11:02
$\begingroup$
$\endgroup$
5
No, this will usually not be possible.
There are very few clusterings that you could use like a classifier. Only with k-means, PAM etc. you could evaluate the "generalization", but clustering has become much more diverse (and interesting) since. And in fact, even the old hierarchical clustering won't generalize well to 'new' data. Clustering isn't classification. Many methods from classification do not transfer well to clustering; including hyperparameter optimization.
If you have only partially labeled data, you can use these labels to optimize parameters. But the general scenario of clustering will be that you want to learn more about your data set; so you run clustering several times, investigate the interesting clusters (because usually, some clusters clearly are too small or too large to be interesting!) and note down some of the insights you got. Clustering is a tool to help the human explore a data set, not a automatic thing. But you will not "deploy" a clustering. They are too unreliable, and a single clustering will never "tell the whole story".
answered Mar 22, 2017 at 7:03
-
1$\begingroup$ Clustering reflects a global property of the data and it commonly has no "ground-truth". Having sad that, I do not think anyone advocates using a clustering as a classifier at first instance; nevertheless if we find an interesting clustering it will be foolish not to try to use the findings by incorporating them in a decision-making process. (Otherwise why did we cluster the data to begin with?) $\endgroup$ Commented Mar 22, 2017 at 11:01
-
$\begingroup$ to run clustering, we still need an objective to optimize. if it is an optimization problem then it can over fit on one data. In addition to kmeans many other methods still need number of clusters. $\endgroup$ Commented Mar 22, 2017 at 13:22
-
1$\begingroup$ Not every clustering algorithm is an optimization problem. $\endgroup$ Commented Mar 23, 2017 at 1:54
-
1$\begingroup$ And as far as using the result: you want to use the insights, not the raw result. Interpret the cluster, and work with the interpretation, because there will be plenty of badly assigned points. $\endgroup$ Commented Mar 23, 2017 at 1:57
-
$\begingroup$ I support this answer, Because when a new data point comes, you learn the representation and then cluster, so there is no need for the test. Even if you are splitting it then you are losing the data information. $\endgroup$ Commented Jan 21, 2020 at 13:10
$\begingroup$
$\endgroup$
4
The sample is not normally split into training and test set in clustering, because, as said in other answers, the test set will not have "true labels" available, so you can't check predictions from the training set on it.
There are however some uses for data set splits in cluster analysis. Particularly one may be interested in whether a clustering structure that is found on one part of the data set corresponds to what goes on in the other half. This isn't quite as easy to formalise though as in supervised classification. We have a paper on this here.
Ullmann, T., Hennig, C., & Boulesteix, A.-L. (2022). Validation of cluster analysis results on validation data: A systematic framework. WIREs Data Mining and Knowledge Discovery, 12( 3), e1444. https://doi.org/10.1002/widm.1444
There is also some work that uses cross-validation, i.e., data set splitting, for estimating the number of clusters, although this is more sophisticated than just using a single training/test split, see, e.g.,
Wang, J. “Consistent Selection of the Number of Clusters via Crossvalidation.” Biometrika 97, no. 4 (2010): 893–904. http://www.jstor.org/stable/29777144,
Fu, W. & Perry, P. O. (2020) Estimating the Number of Clusters Using Cross-Validation, Journal of Computational and Graphical Statistics, 29:1, 162-173, DOI: 10.1080/10618600.2019.1647846
A note on terminology: In some answers you read that "clustering is not classification", but that's a somewhat inappropriate use of terminology. Clustering classifies and can therefore well be called classification, as is done in some literature. A better distinction is between supervised and unsupervised classification, the latter being clustering.
answered May 19, 2023 at 0:05
-
$\begingroup$ Great paper, Christian! Is there a pdf format version of it? $\endgroup$– ttnphnsCommented May 19, 2023 at 14:22
-
$\begingroup$ @ttnphns It's open access and I see a pdf button there. Don't you have that? $\endgroup$ Commented May 19, 2023 at 17:09
-
$\begingroup$ I got it, thanks! The pdf button escapes for me on the mobile view, but I was able to press it on the PC view. $\endgroup$– ttnphnsCommented May 19, 2023 at 18:43
-
$\begingroup$ And I agree with the answer's last paragraph, too. $\endgroup$– ttnphnsCommented May 19, 2023 at 18:44
$\begingroup$
$\endgroup$
Usually you split your data into train and test set when you want to evaluate the performance of your model in the train and test set. In order to do this, you need to know a priori in which cluster every observation belongs to, which means that you know exactly how many clusters there are in your data (equivalently the number of clusters isn't stochastic). Consequently, this means that you have data from let's say k populations. As you can see this isn't a clustering problem but a classification problem.
