2
$\begingroup$

As I read many cases of "standardization",there are some opinions conflict with them, e.g.

  1. some cases will add lag features and some of these features are created by other original features and it might lead to strong correlation between them.however logistics regression models are strongly advised use features with less correlation.

  2. some features are advised to be scaling to (0,1) or (-1,1).if this is it,what's the theory behind it?

Is there a standardization for features handling or for some situations like using certain algorithm or certain feature specialty exists.Or maybe just the final evaluation on test set is the only "standardization " should be concerned,then you could arrange features the way you prefer?

$\endgroup$
0
$\begingroup$

Standardization helps on certain practical performance measures (time till convergence of iterative methods) or making objective functions in certain algorithms reasonable.

For example, consider you are doing clustering using distance based methods (say k-means). Two variables measured on different scale implies that you are measuring a weighted distance between two points and variable which is on larger scale of magnitude will dominate the computation and the results. Similarly, if you are using iterative solver methods for fitting (generalized) linear models, then standardization helps in achieving convergence faster. On the other hand, decision tree based classification/regression methods can handle mix of categorical and non-standardized data as well as missing values out of the box.

So in essence, choice of standardization method is governed by requirements of the algorithm that you want use, even before performance on test set.

$\endgroup$
  • $\begingroup$ how or where could I get these model-building suggestion or standard? $\endgroup$ – george.p Jan 15 '18 at 6:46
  • $\begingroup$ ^ 1. practice on realistic datasets 2. check winning solutions of Kaggle competition. Although Kaggle focuses too much on accuracy (unlike real life), it is possible to learn good amount about feature engineering by studying winning solutions. $\endgroup$ – hssay Jan 15 '18 at 8:34
0
$\begingroup$

You are correct that logistic regression requires features that are uncorrelated in nature. There are different techniques for handling correlation for these cases like - Variable clustering (http://facultyweb.kennesaw.edu/jpriestl/courses/binary/RBS%20PAPER.pdf) followed by variable elimination, variable elimination using Information Value etc. After following these steps, you might still have correlation in the final model variables. To handle such cases, one must compute Variable Inflation Factor (VIF) for all the final variables. As a general rule of thumb no variable should have VIF > 2. If this is the case, remove the variable with the highest VIF, re-run the model and repeat the same steps till all variables have VIF < 2.

An advantage of using standardization is comparing the relative importance of variables in the final model. If all the variables are on the same scale, the variables with the highest coefficients are more potent in predicting your target variables.

$\endgroup$
  • $\begingroup$ how or where could I get these model-building suggestion or standard? $\endgroup$ – george.p Jan 15 '18 at 6:46
  • $\begingroup$ You should talk to experts in the field who have been working on credit risk modelling. They can provide great insights into the standards used by different banks. I have gained my knowledge in a similar way, working on credit risk underwriting for the last 5 years. Another good resource is 'Credit Risk Scorecards by Naeem Siddiqi. Check out the below link: support.sas.com/content/dam/SAS/support/en/books/… $\endgroup$ – kraken Jan 18 '18 at 8:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.