When we study about normalization, various facts are given to explain the necessity.

The most important of all is that:

Normalized column if in higher range than others can have more impact on output and make our results biased.

Simple example: a model which uses features like person age and salary- Age can have low effect on output as its very small and salary might impact more.

But my question is that model should be smart enough to calculate theta as per the range. Age will have higher theta whereas salary will have small. And thus the model will not be biased towards salary.

Another reason they say that in ML, normalization helps algo in converging faster. Similar thing of varied range theta can also be applicable here and hence our algorithm will converge on the same speed.

I need some help in understanding these concepts of normalizing X or coming up with theta with values as per range of X.


One example of the quote you give is the use of standardization in PCA (and thus also PCR): Variables measured on a different scale will by definition have a higher variance, causing PCA to attribute greater importance to such variables. In this case, whether you standardize or not says something about whether variables with greater variance are indeed more 'important', or simply measured on a different scale. A similar argument can be made for normalization in other methods.

Another reason is numerical stability. Very small numbers may be hindered in their precision by the machine precision. Very high numbers may hinder the speed of computation.

As to why algorithms can't take this into consideration by themselves: How else would an algorithm do so other than by normalizing or standardizing?

  • $\begingroup$ I am still not clear on this concept. Sorry could you please explain it more like you explain it to dummies.. $\endgroup$ – Onki May 8 '19 at 4:41
  • $\begingroup$ No need to apologize, but could you be more specific which part you do not understand? $\endgroup$ – Frans Rodenburg May 8 '19 at 4:44
  • $\begingroup$ Could you give me more intution on how greater variance may make model to give more importance to these variables. and How high numbers may hinder the speed of computation. $\endgroup$ – Onki May 8 '19 at 4:57
  • $\begingroup$ if we have respective weight theta then the result would be manged irrespective of the high variance $\endgroup$ – Onki May 8 '19 at 4:58
  • 1
    $\begingroup$ (...) What I meant by speed of computation is that if there are extremely large numbers, then storing them in the memory/multiplying them by other numbers/etc. will also take longer. $\endgroup$ – Frans Rodenburg May 8 '19 at 5:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.