Is cross-validation an effective approach for feature/model selection for microarray data? I've been working with WEKA to build class predictors using this (rather old..) breast cancer dataset.  The dataset is divided into a training and a test set.  I've been testing different learning schemes (mostly focused on feature selection) using 10-Fold cross-validation experiments on the training set.  Unfortunately, when I try the trained models out on the test set there seems to be no correlation whatsoever between scores in cross-validation and actual test set performance.
Is this a consistent problem for microarray or other high dimensional / low sample count data?  Is there another approach that would be more suitable than cross-validation for evaluating models on the training data?
 A: The answer really seems to be that cross-validation is not great because its results are extremely variable but it remains the best option available.  The only other competitive approach seems to be the 0.632 bootstrap estimator which has slightly lower variance but also under-estimates the true performance.  See Is cross-validation valid for small-sample microarray classification?.  Also of relevance - (perhaps obvious) - the more features that are included, the higher the variance of the cv-estimates.
A: I think the problem may be that your training set is too small and therefore not representative of the entire population and if you test it on even smaller tests sets these data can be very different.  This is more of a general large p small n problem and pertains to that type of problemn whether it is genetics or not.  It has nothing to do with how well genes predict outcomes in breast cancer.  In fact I think there are several biomarkers that are useful for estimating the probability of recurrence for patients who had the tumor completely removed.
