I had a discussion with a colleague and we started to wonder, when should one apply feature normalization / scaling to the data? Let's say we have a set of features with some of the features having a very broad range of values and some features having not so broad range of values.
If I'd be doing principal component analysis I would need to normalize the data, this is clear, but let's say we are trying to classify the data by using plain and simple k-nearest neighbour / linear regression method.
Under what conditions should or shouldn't I normalize the data and why? A short and simple example highlighting the point added to the answer would be perfect.