features preprocessing in model building

Question

As I read many cases of "standardization",there are some opinions conflict with them, e.g.

1. some cases will add lag features and some of these features are created by other original features and it might lead to strong correlation between them.however logistics regression models are strongly advised use features with less correlation.

2. some features are advised to be scaling to (0,1) or (-1,1).if this is it,what's the theory behind it?

Is there a standardization for features handling or for some situations like using certain algorithm or certain feature specialty exists.Or maybe just the final evaluation on test set is the only "standardization " should be concerned,then you could arrange features the way you prefer？

Answer 1

Standardization helps on certain practical performance measures (time till convergence of iterative methods) or making objective functions in certain algorithms reasonable.

For example, consider you are doing clustering using distance based methods (say k-means). Two variables measured on different scale implies that you are measuring a weighted distance between two points and variable which is on larger scale of magnitude will dominate the computation and the results. Similarly, if you are using iterative solver methods for fitting (generalized) linear models, then standardization helps in achieving convergence faster. On the other hand, decision tree based classification/regression methods can handle mix of categorical and non-standardized data as well as missing values out of the box.

So in essence, choice of standardization method is governed by requirements of the algorithm that you want use, even before performance on test set.

Answer 2

You are correct that logistic regression requires features that are uncorrelated in nature. There are different techniques for handling correlation for these cases like - Variable clustering (http://facultyweb.kennesaw.edu/jpriestl/courses/binary/RBS%20PAPER.pdf) followed by variable elimination, variable elimination using Information Value etc. After following these steps, you might still have correlation in the final model variables. To handle such cases, one must compute Variable Inflation Factor (VIF) for all the final variables. As a general rule of thumb no variable should have VIF > 2. If this is the case, remove the variable with the highest VIF, re-run the model and repeat the same steps till all variables have VIF < 2.

An advantage of using standardization is comparing the relative importance of variables in the final model. If all the variables are on the same scale, the variables with the highest coefficients are more potent in predicting your target variables.