I have been working with a dataset of 240 medical images of brain tumours.

The dataset is not particularly large but it is heavy, meaning each image is large (in memory) and takes a long time to process.

I want to compare the performance of two segmentation algorithms. These algorithms take a long time to train (2 days already in a GPU) making it painful to perform large experiments such as 10-fold cross-validation.

I was wondering what is the best way to produce some measure of statistical significance for the difference in performance whilst conducting the fewest runs possible.

What I am thinking:

T-tests always recommend using at least 30 trials (n=30). Since my performance metric per case is the percentage of volume overlap (not misclassified or not like in pure classification) can I consider that each individual case in the holdout set is a trial? This would greatly simplify things since it would only require having a holdout set of around 30 cases. Is this valid or statistical nonsense?

  • $\begingroup$ I edited my answer a bit, based on our discussion in the chat. $\endgroup$ – Karolis Koncevičius May 3 '18 at 12:43


Your question is about compaing two machine learning algorithms. In this case there are 240 medical images and you want to compare two different segmentation algorithms that both have to be estimated using a training set. Their performance is measured as "percentage of volume overlap".

General Strategy

The typical thing in such situations in the literature I read is to have the following outline (assuming you want to compare your new method to the existing one):

  1. Provide definition for your new method and reasoning why it is an improvement. This should describe what additional features your method exploits in order to improve the performance compared with it's competitors.
  2. Compare the two methods on some simulated datasets where you control the parameters. These simulated data should be based on the scenarios where your method is considered an improvement.
  3. Compare the methods on real-world dataset examples. This section should demonstrate that the improvements your new methods is incorporating are present in the real world, not only in the simulations.


Since my performance metric per case is the percentage of volume overlap (not misclassified or not like in pure classification) can I consider that each individual case in the holdout set a trial?

In essence the question is - is it valid to compare the performance only using a single training set. The answer depends on the definition of "valid".

In this case (using single training set) the answer we get is only valid for 1) the particular dataset that was used 2) the particular training set that was used. So it's valid, but only with a clarification: "on this particular dataset and using this particular split of training/testing data".

If we have only one dataset we cannot overcome the 1st point. And if our data is only 240 images and we need to train on 210 we cannot really overcome the 2nd point either. Because if we change the training set (210 for training and 30 for testing) our new training set will differ from the last one by only 30 images (and 210 images will be shared by both of them).

However doing at least a few different train/test splits and a paired t-test inside each one of them is still more convincing that only using a single split. In general I would strive for the scenario where each of the images is used (each of them in the end have an assigned performance measure).

The idea is that if different splits consistently show one method beating the other on 30 images then the result seems more convincing. It excludes the possibility of those algorithms being very sensitive to the selection of the training data.


My suggestion would be the following (this assumes you can train the method on 120 training samples):

  1. Split into two parts: 120 training and 120 testing.
  2. Estimate performance measures for all 240 images in the set. (this will require two runs if we split 50/50).
  3. To test if the real difference between algorithm performance is significanyly different from 0 - perform a paired t-test on the images.

This approach will overcome the limitation of only testing on 30 samples and only using one train/test split. Also it would be a lot faster than doing 30 training/testing splits. And if 120 is too small of a sample size to estimate the model - you can instead split into 3 or more parts.

And in order to me more convincing - you can report the results of t-tests separately for each split instead of combining them into one.

  • $\begingroup$ The volume overlap (between 0 and 1) is calculated per image. My doubt is whether in the paired t-test I consider the number of degrees of freedom (n) to be the number of images in the holdout set and the performance metric the individual performance on each image. OR the number of degrees of freedom (n) the number of holdout sets (1 would be too little) and the performance metric the averaged performance of images in the holdout set. $\endgroup$ – Miguel May 3 '18 at 10:37
  • 1
    $\begingroup$ You mean they are not independent because they were "trained" on the same set of images? Yes that is correct. But to be truly independent you would also have to make sure that all "training" sets don't have any images in common. $\endgroup$ – Karolis Koncevičius May 3 '18 at 10:54
  • 1
    $\begingroup$ In order to better mange the sample size you can try looking into Sequential testing. This way if one of the algorithms is clearly better than the other (has constantly better performance values) - you might reach significance before reaching 30 trials. $\endgroup$ – Karolis Koncevičius May 3 '18 at 10:59
  • 1
    $\begingroup$ No no, I didn't say it's valid. I said you are correct, meaning that it's better to do this on 30 different holdout sets. If done on one holdout set of 30 images then your conclusion of one algorithm being better than the other is only valid for the particular training set that was used. $\endgroup$ – Karolis Koncevičius May 3 '18 at 11:03
  • 1
    $\begingroup$ Let us continue this discussion in chat. $\endgroup$ – Karolis Koncevičius May 3 '18 at 11:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.