5
$\begingroup$

For university we need to classify 3 cancer types and give an estimation of how well our model will perform. We received a dataset with 100 samples. We split the data up into a training and test set using stratified sampling with a ratio of 0.3 and 0.7. The resulting training set consists of 69 samples, and the test set out of 31 samples.

We used 10-fold cross validation to calculate the accuracy of our models. When applying the same model on the test set for most models the accuracy on the test set is between 10-15% worse than with cross validation on the training set, except for one model where the accuracy on the test set was 2% better than during cross validation.

The problem that we have now is that the two best scoring models on cross-validation are not significantly different, one has an accuracy of 88.57% +/- 12.45%, the other an accuracy of 88.00% +/- 7.92%. However, on the test set the first score 76%, and the second scores 90%.

If we understood it correctly, we can't choose the second model as the best model based on the test set results, because then we would be using the test set as a training set. Instead, we would like to use repeated cross validation to improve my confidence in the cross validation results, and thereby hopefully being able to choose the best model.

With the small dataset that we have, if we do repeated cross validation and take the average, would we run into the problem that the same folds would be used multiple times?

Improve this question
$\endgroup$
1
  • $\begingroup$ Bit late to the party, but your confidence intervals for the % correctly classified cannot possibly be correct: for 88 % and 88.57% observerd correct classifications with both 69 cases, the confidence intervals must be almost the same. Binomial confidence interval calculation for 61 correct out of 69 cases yields a 95%-c.i. roughly ranging from 79 - 95 %. $\endgroup$
    – cbeleites
    Commented Oct 13, 2015 at 14:15

1 Answer 1

Reset to default
7
$\begingroup$

It seems as if you are using an improper scoring rule, proportion correctly classified. Optimizing this measure will choose a bogus model.

You will need to repeat 10-fold cross-validation 100 times to get sufficient precision for validation estimates, and be sure to use a proper scoring rule (e.g., Brier score (quadratic error score) or logarithmic scoring rule (log likelihood)).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks! Editted: for the assignment we need to give an estimator how many instances we will correctly classify on a blind set. So we need the accuracy. $\endgroup$
    – Niek
    Commented Apr 23, 2013 at 20:12
  • $\begingroup$ How does that make classification the right thing to consider? Just because your instructor mistakenly believes it so? Consider this example: Suppose an outcome occurs in 950 out of 1000 individuals. Classifying every individual as having the outcome, you will be right 0.95 of the time using no data on the subjects. $\endgroup$
    – Frank Harrell
    Commented Apr 23, 2013 at 22:34
  • $\begingroup$ Sadly I don't have any control over what my teachers grade us on. In our full report we use a different measure, but we do need to give an estimation of how many cases we're going to classify correctly. The test set is equally divided (50, 50, 50). $\endgroup$
    – Niek
    Commented Apr 24, 2013 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.