6
$\begingroup$

I've been working with WEKA to build class predictors using this (rather old..) breast cancer dataset. The dataset is divided into a training and a test set. I've been testing different learning schemes (mostly focused on feature selection) using 10-Fold cross-validation experiments on the training set. Unfortunately, when I try the trained models out on the test set there seems to be no correlation whatsoever between scores in cross-validation and actual test set performance.

Is this a consistent problem for microarray or other high dimensional / low sample count data? Is there another approach that would be more suitable than cross-validation for evaluating models on the training data?

Improve this question
$\endgroup$
7

2 Answers 2

Reset to default
2
$\begingroup$

The answer really seems to be that cross-validation is not great because its results are extremely variable but it remains the best option available. The only other competitive approach seems to be the 0.632 bootstrap estimator which has slightly lower variance but also under-estimates the true performance. See Is cross-validation valid for small-sample microarray classification?. Also of relevance - (perhaps obvious) - the more features that are included, the higher the variance of the cv-estimates.

Improve this answer
$\endgroup$
1
$\begingroup$

I think the problem may be that your training set is too small and therefore not representative of the entire population and if you test it on even smaller tests sets these data can be very different. This is more of a general large p small n problem and pertains to that type of problemn whether it is genetics or not. It has nothing to do with how well genes predict outcomes in breast cancer. In fact I think there are several biomarkers that are useful for estimating the probability of recurrence for patients who had the tumor completely removed.

Improve this answer
$\endgroup$
13
  • $\begingroup$ Michael - this is my feeling as well. Could you suggest another approach that handles the large p small n problem more effectively? $\endgroup$
    – Ben
    Commented May 31, 2012 at 19:12
  • $\begingroup$ That is a difficult problem and is not settled statistically. But you should take a look at Efron's empirical Bayes approach which is given in his recent monograph "Large Scale Inference." $\endgroup$
    – Michael R. Chernick
    Commented May 31, 2012 at 19:52
  • $\begingroup$ @Michael, Why do you think it has nothing to do with how well DNA microarray expression data predict breast cancer recurrence? There are no clinically useful microarray-based predictive tests currently used in breast cancer treatment. Currently used markers including the estrogen receptor, progesterone receptor, and the human epidermal growth factor receptor 2 protein are not microarray-based tests. Other markers including Ki-67 and p53 analysis are also not based on the measurement of gene expression levels. $\endgroup$
    – Alexander
    Commented May 31, 2012 at 20:06
  • $\begingroup$ @Alexander I have seen ER and HER2 be effective predictive biomarkers in cancer studies I have worked on at LIMR. I have no idea how they were identified. I assumed it might have been through microarray aalysis. But you probably know better than I do. I have also since a biomarker TIMP-4 that we published a paper on seem to be a good predictor as to whether or not therre would be rapid growth of small area breast cancer tumors. $\endgroup$
    – Michael R. Chernick
    Commented May 31, 2012 at 20:51
  • 1
    $\begingroup$ @Ben I am not sure whether you are addressing your question to me or Alexander. But the point you are making now is exactly the point I thought you made with your initial question. I think that although you may feel that the results are crazy, this is new territory for statistics an area we used to say should not be touched. Statistical dogma always said to model n must be a lot bigger than p. Now we are accepting that something rational can be done when p is a lot larger than n. 20th Century statiticians would be rolling over in their graves! $\endgroup$
    – Michael R. Chernick
    Commented May 31, 2012 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.