I know that k-means is unsupervised and is used for clustering etc and that k-NN is supervised. But I wanted to know concrete differences between the two?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ A concise comparison: web.archive.org/web/20170209125935/http://baoqiang.org/?p=579 $\endgroup$– Franck DernoncourtCommented Oct 30, 2013 at 2:17
-
2$\begingroup$ @FranckDernoncourt The link in your comment to baoqiang.org was broken, so I replaced it with a Wayback Machine link instead. $\endgroup$– Sycorax ♦Commented Aug 7, 2021 at 15:43
Add a comment
|
5 Answers
$\begingroup$
$\endgroup$
5
These are completely different methods. The fact that they both have the letter K in their name is a coincidence.
K-means is a clustering algorithm that tries to partition a set of points into K sets (clusters) such that the points in each cluster tend to be near each other. It is unsupervised because the points have no external classification.
K-nearest neighbors is a classification (or regression) algorithm that in order to determine the classification of a point, combines the classification of the K nearest points. It is supervised because you are trying to classify a point based on the known classification of other points.
-
32$\begingroup$ I think there is more similarity than this guy is giving credit. They both use distance methods to cluster and classify inputs respectively. This is often why they are taught together, and why dimensionality issues are discussed in relation to them. Various distance methods can be applied to both. There are in fact a lot of similarities. $\endgroup$ Commented Oct 16, 2018 at 21:04
-
1$\begingroup$ @eljusticiero67 of course they are used to classify inputs, this is mentioned by OP. And most classical learning methods are distance based, so this is also not surprising. Note that the OP was interested in the differences. Also I understood it as though OP was implying there might be similarity due to the K in both names. $\endgroup$– BitwiseCommented Oct 17, 2018 at 6:57
-
$\begingroup$ "in order to determine the classification of a point, combines the classification of the K nearest points" seems like a recursive definition. How it knows the class of nearest neighbors? I guess in a training stage, the class of training data is gained and in the test time, the class is assigned to a new data using its K nearest neighbors. Again in training and optimization phase it could be similar to clustering... $\endgroup$– AhmadCommented May 29, 2021 at 16:05
-
1$\begingroup$ @Ahmad no, the class of the K nearest points is given as part of the problem, because it is supervised. $\endgroup$– BitwiseCommented Jun 5, 2021 at 10:15
-
$\begingroup$ @Bitwise right I forgot that point! And interestingly, KNN has no training phase! just the label of nodes are used in the test phase. Yet, if one have no training data, maybe k-means for clustering is found useful. $\endgroup$– AhmadCommented Jun 5, 2021 at 10:26
$\begingroup$
$\endgroup$
As noted by Bitwise in their answer, k-means is a clustering algorithm. If it comes to k-nearest neighbours (k-NN) the terminology is a bit fuzzy:
in the context of classification, it is a classification algorithm, as also noted in the aforementioned answer
in general it is a problem, for which various solutions (algorithms) exist
So in the first context, saying "k-NN classifier" can actually mean various underlying concrete algorithms that solve the k-NN problem, and their result is interpreted for the classification purpose.
These are two different things but you might find it interesting that k-means algorithm is one of various possible methods for solving the k-NN problem (Marius Muja and David G. Lowe, "Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration", in International Conference on Computer Vision Theory and Applications (VISAPP'09), 2009 PDF)
$\begingroup$
$\endgroup$
You can have a supervised k-means. You can build centroids (as in k-means) based on your labeled data. Nothing stops you. If you want to improve this, Euclidean space and Euclidean distance might not provide you the best results. You will need to choose your space (could be Riemannian space for example) and define the distance between points (and even define a "point"). The last two are topics of research and they also depend on the type (properties) of data (signal) you have.
answered Jan 11, 2018 at 9:57
$\begingroup$
$\endgroup$
K-means can create the cluster information for neighbour nodes while KNN cannot find the cluster for a given neighbour node.
$\begingroup$
$\endgroup$
k Means can be used as the training phase before knn is deployed in the actual classification stage. K means creates the classes represented by the centroid and class label ofthe samples belonging to each class. knn uses these parameters as well as the k number to classify an unseen new sample and assign it to one of the k classes created by the K means algorithm
