I started a discussion with a collague of mine and we started to wonder, when should one apply feature normalization / scaling to the data? Lets say that we have a set of features with some of the features having a very broad range of values and some features having not so broad range of values.
If I'd be doing principal component analysis I would need to normalize the data, this is clear, but lets say we are trying to classify the data by using plain and simple k-nearest neighbor / linear regression method.
Under what conditions should or shouldn't I normalize the data and why? A short and simple example highlighting the point added to the answer would be perfect.