I've been working with WEKA to build class predictors using this (rather old..) breast cancer dataset. The dataset is divided into a training and a test set. I've been testing different learning schemes (mostly focused on feature selection) using 10-Fold cross-validation experiments on the training set. Unfortunately, when I try the trained models out on the test set there seems to be no correlation whatsoever between scores in cross-validation and actual test set performance.

  1. Is this a consistent problem for microarray or other high dimensional / low sample count data? Is there another approach that would be more suitable than cross-validation for evaluating models on the training data?
  2. Why some of data sets are hard? such as wine.
  • $\begingroup$ I don't understand this sentence: "2.why some of data sets are hard? such as wine." Could you please clarify it? Thx. $\endgroup$ – D L Dahly Mar 5 '14 at 13:06

It is possible that you have made an error of using the CV error as a criterion to select the best gene set during the feature selection -- this way you basically overfitted the model, so no wonder that the test set error is way higher.

If you want to get a proper error estimate of your procedure, you must use nested cross-validation, i.e. add another loop in which you do the feature selection, and only use error estimate from the outer one, completely ignoring the inner one.

Though, well, note that FS methods optimising classification error may not be a best solution.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.