Is it better to constrain the data to a range, say [0,1], or to force a mean of 0 and sd of 1? Why? Does the type of input data matter (I'll be using both continuous and categorical variables)?
$\begingroup$
$\endgroup$
2
-
1$\begingroup$ A more important consideration might be how to scale each variable. Even if all the variables were continuous, I wouldn't necessarily normalize them all the same way -- if the association with the response variable is stronger for x1 than for x2, I'd want to keep the variance on x1 higher than for x2. For example, scale x1 as normal with mean 0 and variance 4, whereas x2 gets variance 1. $\endgroup$– zkurtzCommented Oct 3, 2013 at 11:46
-
$\begingroup$ Well, that's a step away from simplicity and toward potential overfitting that I wouldn't want to take. $\endgroup$– ASOFCommented Oct 3, 2013 at 12:26
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
I think that depends on the data. If you know your feature is bounded, you could scale it to $[0,1]$. If it's binary I guess $\{0,1\}$ is a good choice, perhaps $\{-1,1\}$. Now, if it's unbounded, the standardization to $\text Z$-scores $\overline x = 0$, $\sigma=1$ is a reasonable choice.
$\begingroup$
$\endgroup$
Similar to K-means, KNN uses distance measure. Therefore
- It is better to normalize features. If not, the features with larger values will be dominant.
- If you have too many discrete variables and use dummy coding, distance measures would not work well.
Also, I think my answers for K-means would answer your question of what may happen if we do not normalize features.
answered Sep 9, 2016 at 17:21