3
$\begingroup$

I have a limited size data set of 385 entries on which I want to run multiple classifiers and compare their performance using the WEKA experimenter. The number of attributes in this data set is large, around 190 attributes. Because of the large number of attributes, I want to apply attribute selection.

Because of the limited size of the data set, I do not prefer to split this data set into a separate training set on which I can run the attribute selector a and test set on which the experiment is run using these attributes. Instead, I prefer to use the entire set in a cross-validation experiment using all data set entries. WEKA allows to do the experiment using an AttributeSelectedClassifier in combination with cross-validation.

My question is if it is realy required to perform attribute selection on a separate trainings set or if this setup using the AttributeSelectedClassifier with the entire data set in cross-validation is ok for comparing the performance of multiple classifiers? I.e., is the attribute selection biased when using the AttributeSelectedClassifier in cross-validation mode on my entire set?

$\endgroup$
1
$\begingroup$

I think you really do need to perform attribute selection on data that you do not use to train your model, or else your results will be overly optimistic. The exception is if you're performing attribute selection using unsupervised methods. So it then comes down to how AttributeSelectedClassifier performs feature selection (I'm not familiar with WEKA, so can't help there).

Your example seems to align very well with Hastie's discussion (pgs. 245-247 of ESL of what to be aware of with respect to CV, so I'd take a look and follow his advice.

$\endgroup$
1
$\begingroup$

It's not necessarily true that you'll need to do feature selection for this data. A model like Random Forest will be fine; it can even perform well in cases where you have more features than observations! On the other hand, models in the ordinary linear regression family will likely struggle, so a regularized version (LASSO or elastic net) will likely improve its performance while performing feature selection at the same time.

Your instincts are correct, though: you should not do supervised feature selection and then train a model on that same data. Performance metrics of the model will be very biased upwards. One way to mitigate the effect of this bias would be to perform nested cross-validation, so that at each "outer" CV step, you're exposing the model to new data. This can become expensive as you add CV layers, though.

I don't know how AttributeSelectedClassifier works. A careful study of what it does and a comparison to what you want to do will be the best guide of whether or not it's acceptable for your purposes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.