What is the effect of data scaling when compared between kNN, naive Bayes or logistic regression? I am a complete noob and have just started with machine learning. However there was this one question I faced which I was unable to answer.
What is the effect of data (feature) scaling on the three classifiers, namely kNN, naive Bayes and logistic regression? Would there be any difference between the optimum parameters found by cross validation and evaluation results for scaled vs raw data.
It would be a real great help if some one can explain it graphically.
 A: Each of these ML methods is effected differently by scaling.  Naive Bayes is basically unaffected.  Naive Bayes sets the priors based on the data you give it, so it will scale those priors to match your data.
Logistic regression is in the middle.  If you are doing simple logistic regression, the model can increase or decrease the coefficients on the features as needed to effectively scale the features using the coefficients.  There are two potential negative effects of not scaling though.  First, it may take a lot longer for the model to converge using gradient decent.  Since one unit of movement has a much bigger effect in one feature dimension than the other (see image below) the decent will either bounce around in the smaller dimension and have difficulty hitting the optimum value, or it will crawl slowly along the larger dimension taking a long time to creep up to the optimum.

Second, if you use regularization, it will effect your model.  Since regularization penalizes the size of the coefficients, you will effectively penalize small values more than large values because you need a bigger coefficient on a small value to get the same effect as on a big one.
KNN is going to be effected the most.  Since KNN just looks at the euclidean distance between data points scale matters hugely.  Imagine you have one set of features A with values ranging between 0 and 105 and another B with values ranging from 0 to 1.  A will dominate this model because even if you have values at the maximum possible difference possible between the B values it will only be equivalent to a minuscule difference in A.
