Cross Validation after feature selection

Question

I'm trying to train a decision tree and use cross validation doing so. (this post is quite helpful already understanding CV.)

First I did 10-fold cross validation on my data, including all attributes, with certain parameters. Then, let's say I was happy with the results of the CV, I used the whole data to train a decision tree. Now for example only 4 out of 20 attributes show up in the final decision tree since those 4 attributes already do a good classification job. Should I perform another cross validation with those 4 attributes only? I ask because during each cross validation step random (stratified) samples are taken and everytime another tree might be built. So I might have 10 different trees and average their error rate etc. But isn't the result much more reliable when doing cross validation with those 4 attributes only that eventually show up in the final decision tree instead of using 20 attributes of which some are not relevant for the final tree?

Help is much appreciated!

Answer 1

Now for example only 4 out of 20 attributes show up in the final decision tree since those 4 attributes already do a good classification job. Should I perform another cross validation with those 4 attributes only?

No!

When you did the cross-validation, if you did it properly, you created 10 trees, each one used some subset of the 20 original attributes. You tested each tree on the respective validation set. This way you correctly took the factor "a tree selects only a subset of attributes" into account.

Then you used the whole dataset to create a tree which uses the 4 attributes. So, the whole dataset was used to select these 4 attributes.

Suppose you will try to perform another cross-validation round with 4 attributes. What will happen then? You will create 10 more trees which all use these 4 attributes (or some subset of them), and test them on the validation set, which is the part of the original dataset. So the observations of the validation set, among others, were used for selecting the 4 attributes, and these very observations will be used for testing the model. This is overfitting leakage.