Cross Validation with Preprocessing (Normalization, Discretization, Feature Selection)

Question

I am now trying to evaluate my model with cross validation. My dataset contains some numeric and nominal attributes.

Here, I carry out the following data preprocessing tasks:

A. Normalization: Min-Max Normalization (to [0,1])

B. Discretization: Supervised Discretization (Fayyad-Irani), creating bins with some supervised technique (using class label info)

C. Attribute Selection: Correlation based Feature Selection Method

Actually, I first tried to do preprocessing for the entire set, obtaining preprocessed & reduced dataset, then evaluated my model with the dataset through 10-fold cross validation.

However, I found that the way is commonly (miss)used but not good, optimal because of cheating for the test set.

Hence, I am trying to do preprocessing within cross validation. (Yes, preprocessing for only training set for each fold!)

Here, I have a question. As I know, I think using Normalization Filter obtained from training set should be used for the test set. However, if the range of numeric values in the test set is not covered by that in the training set... Then, the normalization result does not create [0,1] range for the test set. For example, if training set has the range from 30 to 50, but test set has the range from 10 to 100, then the normalization filter obtained from the training set looks not appropriate for the test set.

In this situation, how should I do?

(Plus) Is it acceptable way to do only Normalization and Discretization for the entire set, and do feature selection within the cross validation job?

Thank you in advance! I look forward to receiving very helpful answers! :)

Answer 1

Doing preprocessing out of the cross validation loop is especially bad if feature selection is performed (esp when you have large feature size) but not so much for data normalization, as by scaling either by 1 or 100, these numbers already has a predetermined meaning that there's nothing that the model can cheat and learn about the left-out set.

If you have a problem about this, it reflects more about a programming defect than a mathematical problem. A work around to just to first make the lower and upper bound for your bin incorporate all your data. Yet I don't think packages nowadays have this problem.

Answer 2

As for normalization - if your training samples distribution is different from your validation data distribution then you should expect problems bigger than just using the wrong normalization factors.

Answer 3

So the scaling procedure will still be able to work in this scenario. The sklearn min-max scaler transforms data using two variables derived from the data, namely a scaling factor, equal to 1/range(trainingData) and a shift parameter, equal to -min(trainingData)*scalingFactor. It then scales by multiplying by the scaling factor, and then adding the shift parameter. When it comes to the test data, it again simply multiples by the scaling factor and adds the shift parameter. If the range of the test set is larger than that of the training set, this will naturally produce values less than 0 and larger than 1. Ideally, as @AlexR mentioned we would have the population min and mean to perform our scaling with. However, almost always that information will not be available, and using he sample min and max is often our best estimate.

Though the scaling process can produce values outside of [0, 1], I do not believe that will be too much of an issue in terms of predictive evaluation. Scaling is often a process to ensure that models that are sensitive to the numeric scale of the data perform correctly (EG K-Nearest-Neighbors) and some learning methods might converge quicker with scaled data. In fact, I might guess that it will have so little of an effect that you could scale outside of the CV step and it would not affect your results much (though it would be more correct to include it in the CV step, which I would recommend.) For your other steps, which involve actual modeling and feature selection, it seems that it would be inappropriate to do these outside the feature selection step, as they would be more likely to produce inaccurately optimistic results by cheating on the test set. This mistake has been made with feature selection before; there have been scientific papers that have analyzed microarray data that selected important features outside of the CV process, giving over-optimistic results.