3
$\begingroup$

Doing research on how to approach a problem I'm working on with text data. The gist of online advise I encountered was to cluster my corpus to create labels and then to build a classifier such as e.g. XGBoost based on the newly created labels from clustering.

I sensed it was frowned upon to use the cluster object to classify by determining the closest centroid. I picked up on this after a Google search for the phrase "r predict with kmeans cluster object". Many of the posts recommended flexclust package however the tone I picked up was that you should not use the clustering object to classify e.g. here.

Why is that? Why would it ever be "bad" to use a e.g. a previously built kmeans() cluster object to assign a new datapoint to a cluster by measuring the distance to the closest centroid? As opposed to using a classifier to determine the cluster?

Afterthought since posting. In the context of my current data problem I'm working with text data. It occurred to me that new data might have new tokens/words which would translate into new features and that in this case a classifier might return an error. Whereas, using a previously defined kmeans object would not throw an error, I could still calculate the distance of new data point to the nearest centroid.

In this case surely it's actually better to NOT build a classifier if I want to assign labels to new data where the labels are a previously determined set of clusters.

Improve this question
$\endgroup$
6
  • $\begingroup$ This sort of question was asked a number of times already, if my memory doesn't betray me. Try to search. $\endgroup$
    – ttnphns
    Sep 10, 2017 at 13:08
  • $\begingroup$ It is absolutely all right to assign new objects to clusters/classes based on some proximity function. Actually, it is a particular and the simplest form of classification task because classification also does assigning based on some proximity measure. $\endgroup$
    – ttnphns
    Sep 10, 2017 at 13:16
  • $\begingroup$ (cont.) But here, classes are not investigated to build a separate and special classifier model (which would first learn to discriminate classes). Rather, a proximity descendant from the earlier clustering process or just thought out is being used. Whether this "assign" procedure or "build a classifier then assign" procedure is to be used - is upon your considerations. There are situations when one approach will be more warranred than the other. $\endgroup$
    – ttnphns
    Sep 10, 2017 at 13:16
  • $\begingroup$ Thanks for commenting @ttnphns. Is my final point valid (2nd last paragraph in post)? I've gone the cluster then XGB classifier route. But then I had a thought that new data may contain new text (and thus new features) that the classifier won't recognize. Sticking with the kmeans cluster centroids then seems the "best" approach since I will still be able to assign new clusters based on the previously defined centroids. Does my thinking make sense? $\endgroup$
    – Doug Fir
    Sep 10, 2017 at 13:25
  • 1
    $\begingroup$ The decision is on your. Simple assignment (1) cannot recognize if a new cluster is emerging, (2) cannot handle noise (3) prone to "overfitting" to the specific dataset, (4) does not explicitly model cluster shape. Still, just assigning is a totally valid way to go. $\endgroup$
    – ttnphns
    Sep 10, 2017 at 13:30

2 Answers 2

Reset to default
2
$\begingroup$

Several reasons:

  1. The clusters will not be optimal. When you investigate a cluster, you may be seeing some "pattern", e.g., a cluster appears to be about cars. Then you are tempted to label the cluster as "cars". But there is no guarantee there are only cars in there. If you think "cars" is a good cluster found, you should define a pattern manually that captures only cars.

  2. Some clusters will likely turn out to be garbage. You want to drop them.

  3. You want classes to have meaningful labels, not 0,1,2,3,...

  4. Most classifiers are better at prediction than the nearest centroid classifier (which you would use to reassign new points to k-means centers). Because you can afford to learn more complex models on the final labels, while for clustering when you are still determining the labels iteratively, you need something very fast. So it usually gives better results to train a more advanced classifier once you have decided upon the labels.

  5. K-means assigns points to exactly one class. In real text data, documents will contain 0, 1, or multiple topics. One article may compare cars and bicycles, for example. An others may be just garbage and not belong into any class at all. K-means does not support this.

Improve this answer
$\endgroup$
9
  • 1
    $\begingroup$ Some of these I don't fully understand. However number 4 is interesting. Are you saying that with the newly generated labels one can build a classifier that accounts for new features ("more complex models")? $\endgroup$
    – Doug Fir
    Sep 12, 2017 at 8:58
  • 1
    $\begingroup$ Upvoting, despite I would have comments since the answer sounds vague or repetitive. Pt.2 - agree. Pt.3 - why really? Statistically, to build a classifier or to assign one needs no interpretable labels; so don't agree. (Interpretation is important, it may be an asset as well as a distractor. Sometimes we'll prefer to pospone intrepretation/labeling.) Pt.4 - generally agree with can afford to learn more complex models, but don't get what's "fixed" labels and why we need them (is it yet about interpreted classes, Pt.3?). $\endgroup$
    – ttnphns
    Sep 12, 2017 at 10:02
  • $\begingroup$ (cont.) Pt.5 - true, however, post-K-means assignment (note: the OP was asking not about assignment done during the K-means) could be calibrated to allow for fuze assignment; actually, it is already a simpest form of a classifier. Pt.1 - is again about interpreting of clusters and its role. To repeat what I said: sometimes it is good to leave interpreting out, to focus our classification or assignment purely on statistical properties; it's like "blind experiment". $\endgroup$
    – ttnphns
    Sep 12, 2017 at 10:03
  • $\begingroup$ (cont.) I see your prevailing point in the claim like "make the cluster phenomenon a transcendent class (by investigating it and giving it a label) so it is less accidental and more essential". But that is just a stance or an opinion about what is "valuable". Also, that stance seems to me enough orthogonal to the question "to build a classifier or not to build". $\endgroup$
    – ttnphns
    Sep 12, 2017 at 10:12
  • 1
    $\begingroup$ "fixed" labels = labels that don't change anymore. While training k-means they change, so you need something fast during clustering. You don't want to train a random forest for every iteration of k-means to label points during clustering instead of nearest-centroid. I'll edit this to "final". $\endgroup$
    – Has QUIT--Anony-Mousse
    Sep 12, 2017 at 14:25
1
$\begingroup$

Because assigning a new data point to one of the 'classes' is a classification work, rather than clustering.

Let me elaborate more specifically.

  1. If you have $n$ items and you want to form clusters in which similar (or close) items are included, you can carry out clustering analysis.
  2. However, if you already have $n$ items clustered (or classified), and you want to assing the $n+1$ th item to one of the clusters (or classes), it is better to perform classification.

Clustering and classification are rather confusing concepts. Be sure to clarify the differences between them. Refer to here for the differences.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thanks for sharing your advice. I understand the difference between the two. but if it's a simple matter of assigning a new data point based on the previously defined clusters I could either calculate the distance to the nearest centroid (100% accuracy) or build a classifier (high acuracy but still, why not just calculate to the centroid?) $\endgroup$
    – Doug Fir
    Sep 10, 2017 at 12:31
  • $\begingroup$ @DougFir Yes, you can calculate the distance from each centroid. But that is also a classification, rather than clustering. You are "classifying" the new data point to one of the clusters based on the distance towards the centroids already determined. $\endgroup$
    – Buomsoo Kim
    Sep 10, 2017 at 13:04
  • $\begingroup$ Right. However this seems to be frowned upon... I think. Is there anything 'wrong" with doing this on new data? $\endgroup$
    – Doug Fir
    Sep 10, 2017 at 13:04
  • $\begingroup$ @DougFir I do not think it is anything "wrong". It is understanding the overall data structure with unsupervised learning (i.e., clustering) first and applying supervised learning (i.e., classification) on top of it. This sort of problem solving is often utilized when you do not have enough labeled data. $\endgroup$
    – Buomsoo Kim
    Sep 10, 2017 at 13:07
  • $\begingroup$ Actually, this answer does not answers it, it doesn't explain why beyond telling definitions of the concepts. $\endgroup$
    – ttnphns
    Sep 10, 2017 at 13:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.