2
$\begingroup$

I know that it is good practice to perform normalization (subtracting the data by its mean and dividing it by its standard deviation) first on the training data, and in a later step to use the mean and standard deviation of the training data to normalize also the test data.

I am aware from feature selection, that the feature selection for supervised feature selection methods - i.e. feature selection methods which do make use of the class labels - must be done solely based on the training set. However, since normalization does not make use of any class labels, I wonder how the common practice described above is justified?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Pegah,

Really all feature selection should be "trained" on the training set alone, and then applied to the test set. The purpose of the testing procedure is to simulate what will happen when you train your model, and then begin comparing your predictions to data you have not seen. So the idea is to do your entire model training procedure without looking at the test set. This gives you the purest estimation of generalization error.

For example in this specific case imagine we are doing feature normalization before KNN-regression. The extent to which a certain variable varies in your test set, will be taken into account in the normalization, which will have an effect on the distance metric used for KNN. So you are doing some learning from your test set, which is not ideal.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ That goes against the literature, see for instance Pereira et. al (Machine Learning and fmri) or Smialowski et al. (Pitfalls of supervised feature selection) and many more sources which all state that this rule only applies to supervised feature selection. $\endgroup$
    – Pugl
    Sep 29, 2015 at 17:30
  • $\begingroup$ That is also stated in "Elements of Statistical Learning" (Tibrishani et al.): "There is one qualification: initial unsupervised screening steps can be done before samples are left out." (page 246), so no, it is not generally true $\endgroup$
    – Pugl
    Sep 29, 2015 at 17:45
  • $\begingroup$ When I am not at work I will see if I can get my hands on these. Do any of them say why this is does not bias the estimation of generalization error? I suppose one argument is that in the future if you are asked to predict $y$ from $x$, you have $x$ so you can technically refrain the feature selection. This fact does not apply in my application, but this could be the reason. I tend to look at testing as a way of testing an entire algorithm to estimate future performance, so I hold everything out. $\endgroup$
    – jlimahaverford
    Sep 29, 2015 at 18:38
  • $\begingroup$ They do not go into much detail. But it makes sense to me that if you use say a crude thresholding (only keeping features above a certain value) you do not bias the performance assessment, since no class information is contained in the thresholding..thinking about it though, even thresholding might alter classification accuracy if e.g. one class contains disproportionately more values above a given threshold, so hmmm...not sure anymore $\endgroup$
    – Pugl
    Sep 29, 2015 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.