I have a limited size data set of 385 entries on which I want to run multiple classifiers and compare their performance using the WEKA experimenter. The number of attributes in this data set is large, around 190 attributes. Because of the large number of attributes, I want to apply attribute selection.
Because of the limited size of the data set, I do not prefer to split this data set into a separate training set on which I can run the attribute selector a and test set on which the experiment is run using these attributes. Instead, I prefer to use the entire set in a cross-validation experiment using all data set entries. WEKA allows to do the experiment using an AttributeSelectedClassifier in combination with cross-validation.
My question is if it is realy required to perform attribute selection on a separate trainings set or if this setup using the AttributeSelectedClassifier with the entire data set in cross-validation is ok for comparing the performance of multiple classifiers? I.e., is the attribute selection biased when using the AttributeSelectedClassifier in cross-validation mode on my entire set?