31
$\begingroup$

Could someone please explain to me why you need to normalize data when using K nearest neighbors.

I've tried to look this up, but I still can't seem to understand it.

I found the following link:

https://discuss.analyticsvidhya.com/t/why-it-is-necessary-to-normalize-in-knn/2715

But in this explanation, I don't understand why a larger range in one of the features affects the predictions.

Improve this question
$\endgroup$
3
  • $\begingroup$ I think normalization has to be justified from the subject-matter point of view. Essentially, what matters is what defines the distance between points. You have to find a convenient arithmetic definition of distance that reflects the subject-matter definition of distance. In my limited experience, I have normalize in some but not all directions based on subject-matter considerations. $\endgroup$
    – Richard Hardy
    Commented Jun 26, 2017 at 19:00
  • 1
    $\begingroup$ For an instructive example, please see stats.stackexchange.com/questions/140711. $\endgroup$
    – whuber ♦
    Commented Jun 26, 2017 at 19:28
  • $\begingroup$ It seems like any scaling (min-max or robust) is acceptable, not just standard scaling. Is that correct? $\endgroup$
    – skeller88
    Commented Apr 10, 2020 at 20:20

4 Answers 4

Reset to default
45
$\begingroup$

The k-nearest neighbor algorithm relies on majority voting based on class membership of 'k' nearest samples for a given test point. The nearness of samples is typically based on Euclidean distance.

Consider a simple two class classification problem, where a Class 1 sample is chosen (black) along with it's 10-nearest neighbors (filled green). In the first figure, data is not normalized, whereas in the second one it is.

Data without normalization Data with normalization

Notice, how without normalization, all the nearest neighbors are aligned in the direction of the axis with the smaller range, i.e. $x_1$ leading to incorrect classification.

Normalization solves this problem!

Improve this answer
$\endgroup$
7
  • 4
    $\begingroup$ This answer is exactly right, but I fear the illustrations might be deceptive because of the distortions involved. The point might be better made by drawing them both so that the two axes in each are at the same scale. $\endgroup$
    – whuber ♦
    Commented Jun 26, 2017 at 19:30
  • 1
    $\begingroup$ I found it difficult to fit all data points in the same scale for both figures. Hence, I mentioned in a note that scales of axes are different. $\endgroup$
    – kedarps
    Commented Jun 26, 2017 at 19:55
  • 1
    $\begingroup$ That difficulty actually is the point of your response! One way to overcome it is not to use such an extreme range of scales. A 5:1 difference in scales, rather than a 1000:1 difference, would still make your point nicely. Another way is to draw the picture faithfully: the top scatterplot will seem to be a vertical line of points. $\endgroup$
    – whuber ♦
    Commented Jun 26, 2017 at 19:57
  • 2
    $\begingroup$ @whuber, I misunderstood your first comment. Fixed the plots, hopefully it's better now! $\endgroup$
    – kedarps
    Commented Jun 26, 2017 at 20:10
  • 1
    $\begingroup$ @Undertherainbow That is correct! $\endgroup$
    – kedarps
    Commented Mar 11, 2019 at 19:33
10
$\begingroup$

Suppose you had a dataset (m "examples" by n "features") and all but one feature dimension had values strictly between 0 and 1, while a single feature dimension had values that range from -1000000 to 1000000. When taking the euclidean distance between pairs of "examples", the values of the feature dimensions that range between 0 and 1 may become uninformative and the algorithm would essentially rely on the single dimension whose values are substantially larger. Just work out some example euclidean distance calculations and you can understand how the scale affects the nearest neighbor computation.

Improve this answer
$\endgroup$
0
0
$\begingroup$

If the scale of features is very different then normalization is required. This is because the distance calculation done in KNN uses feature values. When the one feature values are large than other, that feature will dominate the distance hence the outcome of the KNN.

see example on gist.github.com

Improve this answer
$\endgroup$
0
$\begingroup$

The larger the scale a particular feature has relative to other features, the more weight that feature will have in distance calculations. Scaling all features to a common scale gives each feature an equal weight in distance calculations. But notice that scaling introduces a particular weighting on the distance function, so how can we assume that it is somehow the correct one for KNN? So my answer is: scaling should not be assumed to be a requirement.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.