Wiki gives this definition of KNN

In pattern recognition, the k-nearest neighbors algorithm (k-NN) is a non-parametric method used for classification and regression. In both cases, the input consists of the k closest training examples in the feature space. The output depends on whether k-NN is used for classification or regression:

  • In k-NN classification, the output is a class membership. An object is classified by a plurality vote of its neighbors, with the object
    being assigned to the class most common among its k nearest neighbors (k is a positive integer, typically small). If k = 1, then the object is simply assigned to the class of that single nearest neighbor.
  • In k-NN regression, the output is the property value for the object. This value is the average of the values of k nearest neighbors.

k-NN is a type of instance-based learning, or lazy learning, where the function is only approximated locally and all computation is deferred until classification.

Does "locally" here mean only top K instances contribute the prediction? If yes, global means all instances contribute the prediction? It seems that except for knn, all other machine learning do the prediction globally, is my understanding right?


1 Answer 1


They mean that $k$-NN does not directly learn any functional form of the description of the data, but instead during prediction time it returns a sample ($k=1$), or average of the samples ($k>1$), that are closest to your test example. It does not have any "global" knowledge of the function, but can only give you the "local" examples, that are samples from the underlying function.

  • $\begingroup$ Thanks for your answer. Would you please give some concrete examples about "global" knowledge of the function, such as ${\displaystyle h_{i}=\mathbf {x} _{i}^{\mathsf {T}}{\boldsymbol {\beta }}+\varepsilon _{i}.}$ for linear regression? $\endgroup$
    – JJJohn
    Commented Aug 18, 2019 at 21:51
  • $\begingroup$ @fuDL you seem to have answered yourself. $\endgroup$
    – Tim
    Commented Aug 19, 2019 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.