I've been working with WEKA to build class predictors using this (rather old..) breast cancer dataset. The dataset is divided into a training and a test set. I've been testing different learning schemes (mostly focused on feature selection) using 10-Fold cross-validation experiments on the training set. Unfortunately, when I try the trained models out on the test set there seems to be no correlation whatsoever between scores in cross-validation and actual test set performance.
Is this a consistent problem for microarray or other high dimensional / low sample count data? Is there another approach that would be more suitable than cross-validation for evaluating models on the training data?