Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 5 down vote favorite
2

I've been working with WEKA to build class predictors using this (rather old..) breast cancer dataset. The dataset is divided into a training and a test set. I've been testing different learning schemes (mostly focused on feature selection) using 10-Fold cross-validation experiments on the training set. Unfortunately, when I try the trained models out on the test set there seems to be no correlation whatsoever between scores in cross-validation and actual test set performance.

Is this a consistent problem for microarray or other high dimensional / low sample count data? Is there another approach that would be more suitable than cross-validation for evaluating models on the training data?

share|improve this question
I think the real problem is that gene expression profiling does not, in fact, actually predict the clinical outcome in breast cancer. – Alexander May 31 '12 at 17:36
Are you sure you have included FS/MS in your CV loop? Making FS once and testing its output with CV is not enough and is a straight way to overfitting. – mbq♦ May 31 '12 at 18:56
The feature selection routines are included in the CV loop. (In WEKA I use the AttributeSelectedClassifier and then test different selection approaches.) – Ben May 31 '12 at 19:10
2  
Also, you might be interested in the paper Using cross-validation to evaluate predictive accuracy of survival risk classifiers based on high-dimensional data. Gene Expression–Based Prognostic Signatures in Lung Cancer: Ready for Clinical Use? would also be of interest to you. Both provide really wonderful discussions of this topic, although if I recall they don't provide full discussions of intra-tumor heterogeneity. – Alexander May 31 '12 at 20:58
2  
May be this tutorial could help you cbio.mskcc.org/~lianos/tips/svms-and-gene-signatures – friveroll Jun 1 '12 at 1:31
show 2 more comments

2 Answers

up vote 1 down vote

The answer really seems to be that cross-validation is not great because its results are extremely variable but it remains the best option available. The only other competitive approach seems to be the 0.632 bootstrap estimator which has slightly lower variance but also under-estimates the true performance. See Is cross-validation valid for small-sample microarray classification?. Also of relevance - (perhaps obvious) - the more features that are included, the higher the variance of the cv-estimates.

share|improve this answer
up vote 0 down vote

I think the problem may be that your training set is too small and therefore not representative of the entire population and if you test it on even smaller tests sets these data can be very different. This is more of a general large p small n problem and pertains to that type of problemn whether it is genetics or not. It has nothing to do with how well genes predict outcomes in breast cancer. In fact I think there are several biomarkers that are useful for estimating the probability of recurrence for patients who had the tumor completely removed.

share|improve this answer
Michael - this is my feeling as well. Could you suggest another approach that handles the large p small n problem more effectively? – Ben May 31 '12 at 19:12
That is a difficult problem and is not settled statistically. But you should take a look at Efron's empirical Bayes approach which is given in his recent monograph "Large Scale Inference." – Michael Chernick May 31 '12 at 19:52
@Michael, Why do you think it has nothing to do with how well DNA microarray expression data predict breast cancer recurrence? There are no clinically useful microarray-based predictive tests currently used in breast cancer treatment. Currently used markers including the estrogen receptor, progesterone receptor, and the human epidermal growth factor receptor 2 protein are not microarray-based tests. Other markers including Ki-67 and p53 analysis are also not based on the measurement of gene expression levels. – Alexander May 31 '12 at 20:06
@Alexander I have seen ER and HER2 be effective predictive biomarkers in cancer studies I have worked on at LIMR. I have no idea how they were identified. I assumed it might have been through microarray aalysis. But you probably know better than I do. I have also since a biomarker TIMP-4 that we published a paper on seem to be a good predictor as to whether or not therre would be rapid growth of small area breast cancer tumors. – Michael Chernick May 31 '12 at 20:51
1  
@Ben I am not sure whether you are addressing your question to me or Alexander. But the point you are making now is exactly the point I thought you made with your initial question. I think that although you may feel that the results are crazy, this is new territory for statistics an area we used to say should not be touched. Statistical dogma always said to model n must be a lot bigger than p. Now we are accepting that something rational can be done when p is a lot larger than n. 20th Century statiticians would be rolling over in their graves! – Michael Chernick May 31 '12 at 23:03
show 8 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.