Test if performance difference of two image segmentation algorithms is statistically significant I have been working with a dataset of 240 medical images of brain tumours.
The dataset is not particularly large but it is heavy, meaning each image is large (in memory) and takes a long time to process.
I want to compare the performance of two segmentation algorithms. These algorithms take a long time to train (2 days already in a GPU) making it painful to perform large experiments such as 10-fold cross-validation.
I was wondering what is the best way to produce some measure of statistical significance for the difference in performance whilst conducting the fewest runs possible.
What I am thinking:
T-tests always recommend using at least 30 trials (n=30).
Since my performance metric per case is the percentage of volume overlap (not misclassified or not like in pure classification) can I consider that each individual case in the holdout set is a trial? This would greatly simplify things since it would only require having a holdout set of around 30 cases. Is this valid or statistical nonsense?
 A: Situation
Your question is about compaing two machine learning algorithms. In this case there are 240 medical images and you want to compare two different segmentation algorithms that both have to be estimated using a training set. Their performance is measured as "percentage of volume overlap".

General Strategy
The typical thing in such situations in the literature I read is to have the following outline (assuming you want to compare your new method to the existing one):


*

*Provide definition for your new method and reasoning why it is an improvement. This should describe what additional features your method exploits in order to improve the performance compared with it's competitors.

*Compare the two methods on some simulated datasets where you control the parameters. These simulated data should be based on the scenarios where your method is considered an improvement.

*Compare the methods on real-world dataset examples. This section should demonstrate that the improvements your new methods is incorporating are present in the real world, not only in the simulations.



Question

Since my performance metric per case is the percentage of volume overlap (not misclassified or not like in pure classification) can I consider that each individual case in the holdout set a trial?

In essence the question is - is it valid to compare the performance only using a single training set. The answer depends on the definition of "valid".
In this case (using single training set) the answer we get is only valid for 1) the particular dataset that was used 2) the particular training set that was used. So it's valid, but only with a clarification: "on this particular dataset and using this particular split of training/testing data".
If we have only one dataset we cannot overcome the 1st point. And if our data is only 240 images and we need to train on 210 we cannot really overcome the 2nd point either. Because if we change the training set (210 for training and 30 for testing) our new training set will differ from the last one by only 30 images (and 210 images will be shared by both of them).
However doing at least a few different train/test splits and a paired t-test inside each one of them is still more convincing that only using a single split. In general I would strive for the scenario where each of the images is used (each of them in the end have an assigned performance measure).
The idea is that if different splits consistently show one method beating the other on 30 images then the result seems more convincing. It excludes the possibility of those algorithms being very sensitive to the selection of the training data.

Suggestion
My suggestion would be the following (this assumes you can train the method on 120 training samples):


*

*Split into two parts: 120 training and 120 testing.

*Estimate performance measures for all 240 images in the set. (this will require two runs if we split 50/50).

*To test if the real difference between algorithm performance is significanyly different from 0 - perform a paired t-test on the images.


This approach will overcome the limitation of only testing on 30 samples and only using one train/test split. Also it would be a lot faster than doing 30 training/testing splits. And if 120 is too small of a sample size to estimate the model - you can instead split into 3 or more parts.
And in order to me more convincing - you can report the results of t-tests separately for each split instead of combining them into one.
