Return to Answer

Consider this example. We generate some target data by flipping a coin 10 times and recording whether it comes down as heads or tails. Next we generate 20 features by flipping the coin 10 times for each feature and write down what we get. We then perform feature selection by picking the feature that matches the target data as closely as possible and use that as our prediction. If we then cross-validate, we will get an expected error rate slightly lower than 0.5. This is because we have chosen the feature on the basis of a correlation over both the training set and the test set in every fold of the cross-validation procedure. However the true error rate is going to be 0.5 as the target data is simply random. If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct).

The key idea is that cross-validation is a way of estimating the generalisation performance of a process for building a model, so you need to repeat the whole process in each fold. Otherwise you will end up with a biased estimate, or an under-estimate of the variance of the estimate (or both).

Here is some MATLAB code that performs a Monte-Carlo simulation of this set up, with 56 features and 259 cases, to match your example, the output it gives is:

Consider this example. We generate some target data by flipping a coin 10 times and recording whether it comes down as heads or tails. Next, we generate 20 features by flipping the coin 10 times for each feature and write down what we get. We then perform feature selection by picking the feature that matches the target data as closely as possible and use that as our prediction. If we then cross-validate, we will get an expected error rate slightly lower than 0.5. This is because we have chosen the feature on the basis of a correlation over both the training set and the test set in every fold of the cross-validation procedure. However, the true error rate is going to be 0.5 as the target data is simply random. If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct).

The key idea is that cross-validation is a way of estimating the generalization performance of a process for building a model, so you need to repeat the whole process in each fold. Otherwise, you will end up with a biased estimate, or an under-estimate of the variance of the estimate (or both).

Here is some MATLAB code that performs a Monte-Carlo simulation of this setup, with 56 features and 259 cases, to match your example, the output it gives is:

Consider this example. We generate some target data by flipping a coin 10 times and recording whether it comes down as heads or tails. Next we generate 20 features by flipping the coin 10 times for each feature and write down what we get. We then perform feature selection by picking the feature that matches the target data as closely as possible and use that as our prediction. If we then cross-validate, we will get an expected error rate slightly lower than 0.5. This is because we have chosen the feature on the basis of a correllation over both the training set and the test set in every fold of the cross-validation procedure. However the true error rate is going to be 0.5 as the target data is simply random. If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct).

The biased estimator is the one where feature selection is performed prior to cross-validation, the unbiased estimator is the one where feature selection is performed independendtly in each fold of the cross-validation. This suggests that the bias can be quite severe in this case, depending on the nature of the learning task.

Consider this example. We generate some target data by flipping a coin 10 times and recording whether it comes down as heads or tails. Next we generate 20 features by flipping the coin 10 times for each feature and write down what we get. We then perform feature selection by picking the feature that matches the target data as closely as possible and use that as our prediction. If we then cross-validate, we will get an expected error rate slightly lower than 0.5. This is because we have chosen the feature on the basis of a correlation over both the training set and the test set in every fold of the cross-validation procedure. However the true error rate is going to be 0.5 as the target data is simply random. If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct).

The biased estimator is the one where feature selection is performed prior to cross-validation, the unbiased estimator is the one where feature selection is performed independently in each fold of the cross-validation. This suggests that the bias can be quite severe in this case, depending on the nature of the learning task.

Consider this example. We generate some target data by flipping a coin 10 times and recording whether it comes down as heads or tails. Next we generate 20 features by flipping the coin 10 times for each feature and write down what we get. We then perform feature selection by picking the attribute that matches the target data as closely as possible and use that as out prediction. If we then cross-validate, we will get an expected error rate slightly lower than 0.5. This is because we have chosen the feature on the basis of a correllation over both the training set and the test set in every fold of the cross-validation procedure. However the true error rate is going to be 0.5 as the target data is simply random. If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct).

HTH

Which suggests that the bias can be quite severe in this case, depending on the nature of the learning task.

Consider this example. We generate some target data by flipping a coin 10 times and recording whether it comes down as heads or tails. Next we generate 20 features by flipping the coin 10 times for each feature and write down what we get. We then perform feature selection by picking the feature that matches the target data as closely as possible and use that as our prediction. If we then cross-validate, we will get an expected error rate slightly lower than 0.5. This is because we have chosen the feature on the basis of a correllation over both the training set and the test set in every fold of the cross-validation procedure. However the true error rate is going to be 0.5 as the target data is simply random. If you perform feature selection independently within each fold of the cross-validation, the expected value of the error rate is 0.5 (which is correct).

The key idea is that cross-validation is a way of estimating the generalisation performance of a process for building a model, so you need to repeat the whole process in each fold. Otherwise you will end up with a biased estimate, or an under-estimate of the variance of the estimate (or both).

HTH

The biased estimator is the one where feature selection is performed prior to cross-validation, the unbiased estimator is the one where feature selection is performed independendtly in each fold of the cross-validation. This suggests that the bias can be quite severe in this case, depending on the nature of the learning task.