14
$\begingroup$

I know that there is more than two type of normalizing.

For example,

1- Transforming data using a z-score or t-score. This is usually called standardization.

2- Rescaling data to have values between 0 and 1.

The question now if I need normalizing

Which type of data normalizing should be used with KNN? and Why?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
16
$\begingroup$

For k-NN, I'd suggest normalizing the data between $0$ and $1$.

k-NN uses the Euclidean distance, as its means of comparing examples. To calculate the distance between two points $x_1 = (f_1^1, f_1^2, ..., f_1^M)$ and $x_2 = (f_2^1, f_2^2, ..., f_2^M)$, where $f_1^i$ is the value of the $i$-th feature of $x_1$:

$$ d(x_1, x_2) = \sqrt{(f_1^1 - f_2^1)^2 + (f_1^2 - f_2^2)^2 + ... + (f_1^M - f_2^M)^2} $$

In order for all of the features to be of equal importance when calculating the distance, the features must have the same range of values. This is only achievable through normalization.

If they were not normalized and for instance feature $f^1$ had a range of values in $[0, 1$), while $f^2$ had a range of values in $[1, 10)$. When calculating the distance, the second term would be $10$ times important than the first, leading k-NN to rely more on the second feature than the first. Normalization ensures that all features are mapped to the same range of values.

Standardization, on the other hand, does have many useful properties, but can't ensure that the features are mapped to the same range. While standardization may be best suited for other classifiers, this is not the case for k-NN or any other distance-based classifier.

Improve this answer
$\endgroup$
3
  • 4
    $\begingroup$ Is your answer will be the same if I used different distance instead of Euclidean distance (for example Manhattan distance or other distance even fractional distance)? Also If the range of the variables is approximately close to each other. $\endgroup$
    – jeza
    Commented Aug 25, 2018 at 12:33
  • 8
    $\begingroup$ Yes I just showed Euclidean distance as an example, but all distance metrics suffer from the same thing. If the ranges are close to one another then it wouldn't affect the calculation of the metric that much, but it still would. For example if $f^1 \in [0, 1)$ and $f^2 \in [0, 1.2)$, $f^2$ would still be $20\%$ more important than $f^1$. One thing I forgot to mention was that standardizing, obviously, is much better than not performing any feature scaling; it is simply worse than normalization. $\endgroup$
    – Djib2011
    Commented Aug 25, 2018 at 13:07
  • $\begingroup$ Ah I see. "it is simply worse than normalization"!? $\endgroup$
    – jeza
    Commented Aug 25, 2018 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.