I always read in books that when we do classification or machine learning tasks it's always better to normalize the features so to make them in one range like 0-1. Today I used weka to play with Iris dataset. First I just built a J48 classifier without normalizing the values, and the it made perfect performance. However when I normalized all the features to be in the range 0-1, the classifier made so much mistakes. Why is that? Shouldn't normalization be used always?
$\begingroup$
$\endgroup$
3
-
$\begingroup$ The context is too broad, and you did not state why you would want to optimize an improper accuracy scoring rule such as classification accuracy. You also need to state what problem normalization would attempt to solve. In general normalization gets in the way of understanding more than it helps, but occasionally normalization reduces the dimensionality of the model in a helpful way. $\endgroup$– Frank HarrellOct 31, 2014 at 12:58
-
$\begingroup$ Note that normalization means different things within statistical science. The tag's definition is scaling to $[0,1]$; yours may differ. $\endgroup$– Nick CoxOct 31, 2014 at 15:46
-
$\begingroup$ Are you measuring accuracy on the same data you trained on, or on a held-out test set? And how did you normalize down to [0,1]? Both of these can make a difference in your results. $\endgroup$– WayneSep 19, 2016 at 14:48
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
1
If your attributes already have a meaningful and comparable scale then normalization can destroy important information.
Take e.g. data coming from a physical experiment. Coordinates are measure in x,y,z, each axis is in milimeters. Since the experiment is performed on a flat dish, x and y vary on the range of 0-100 (i.e. 10 centimeters), but the z axis only varies from 0-10 (i.e. a 1 cm high box).
Normalizing such data with greatly emphasize the z axis, which most likely is not supported by a physical interpretation of the results.
Key point of the story: understanding your data is essential.
Normalization is a hotfix if you don't understand the scales of your data.
answered Nov 1, 2014 at 12:02
-
$\begingroup$ Thanks a lot. I have been thinking about cases where normalization could hurt. Your example makes sense. I could see the performance degrades with normalization, especially if the collected data does not have a uniform distribution across x-axis due to some limitations of the machine. $\endgroup$– SaTaJan 20, 2021 at 3:06
$\begingroup$
$\endgroup$
It depends on the algorithm. For some algorithms normalization has no effect. Generally, algorithms that work with distances tend to work better on normalized data but this doesn't mean the performance will always be higher after normalization.
Note that many algorithms have tuning parameters which you may need to change after normalization. What you are seeing may just be that the default parameter settings for J48 happened to work well for the unnormalized data.
answered Jan 19, 2014 at 10:04
$\begingroup$
$\endgroup$
Typically decision trees (for instance C4.5, implemented as J48 in Weka you used) are non parametric, that is they don't make any assumption regarding the distribution of the data. As long as the normalization doesn't change the ranks of the data (and I know of no normalization that does that), the results will be exactly the same (you will only get different splitting levels).
Of course this doesn't hold for algorithms making parametric assumptions (logistic regression, etc.) So you shouldn't always normalize, but you should decide to do it or not depending on your algorithm.
