Imagine you have two different datasets each having a feature representing people's ages. One dataset is gathered from teenagers the other is from elderly people.

# Feature 1:
ages_1 = [15, 15, 16, 17, 17, 18, 19, 19]

# Feature 2:
ages_2 = [75, 75, 76, 77, 77, 78, 79, 79]

If we scale these two features, we'll get exactly the same vector, meaning that we'll lose the information that they represent two completely different age groups. Furthermore, unless we store the parameters of the transformation (min/max or mean/std), this information will be irretrievable.

Another consequence of feature scaling is that features lose their interpretability. For example an age of 0.87 (after scaling) would just mean that it is belongs to one of the oldest people in the dataset (could be 18 in the first case or 78 in the second - there is no way we can tell).

Given that we have a lot to lose by scaling the features, why is feature scaling so popular in Machine Learning?

  • $\begingroup$ I guess when you have multiple features and you want to compare them but they are on different units of measurement, you need to standardize to make them comparable. take e.g. penalized regression where you give the same weight to all regression coefficients. in this case you need scaled variables otherwise your penalty will have different effect based on the unit of measurement $\endgroup$
    – mkln
    Sep 17 '18 at 19:53

Some Machine Learning algorithms require all features to be in the same range to function properly, or they'll tend to pay more attention to some features rather than the other. An example of such algorithms are distance-based algorithms.

For example say you have a dataset where two features are:

age = [33, 35, 55, 67, 77, 78, 80, 83, 85, 93]
height = [1.67, 1.72, 1.73, 1.76, 1.8, 1.81, 1.83, 1.85, 1.88, 1.91]

The difference from the tallest to the shortest person is just 0.24 units, while the difference from the oldest to the youngest is 60 units. This means that this algorithm will treat the ages as far more important than the heights.

By normalizing the features to the same distance, you are ensuring that the algorithm treats them with equal importance.

You can also read this relevant post, with a more detailed answer on why normalization is required for k-NN.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.