Your question is about compaing two machine learning algorithms. In this case there are 240 medical images and you want to compare two different segmentation algorithms that both have to be estimated using a training set. Their performance is measured as "percentage of volume overlap".
The typical thing in such situations in the literature I read is to have the following outline (assuming you want to compare your new method to the existing one):
- Provide definition for your new method and reasoning why it is an improvement. This should describe what additional features your method exploits in order to improve the performance compared with it's competitors.
- Compare the two methods on some simulated datasets where you control the parameters. These simulated data should be based on the scenarios where your method is considered an improvement.
- Compare the methods on real-world dataset examples. This section should demonstrate that the improvements your new methods is incorporating are present in the real world, not only in the simulations.
Since my performance metric per case is the percentage of volume overlap (not misclassified or not like in pure classification) can I consider that each individual case in the holdout set a trial?
In essence the question is - is it valid to compare the performance only using a single training set. The answer depends on the definition of "valid".
In this case (using single training set) the answer we get is only valid for 1) the particular dataset that was used 2) the particular training set that was used. So it's valid, but only with a clarification: "on this particular dataset and using this particular split of training/testing data".
If we have only one dataset we cannot overcome the 1st point. And if our data is only 240 images and we need to train on 210 we cannot really overcome the 2nd point either. Because if we change the training set (210 for training and 30 for testing) our new training set will differ from the last one by only 30 images (and 210 images will be shared by both of them).
However doing at least a few different train/test splits and a paired t-test inside each one of them is still more convincing that only using a single split. In general I would strive for the scenario where each of the images is used (each of them in the end have an assigned performance measure).
The idea is that if different splits consistently show one method beating the other on 30 images then the result seems more convincing. It excludes the possibility of those algorithms being very sensitive to the selection of the training data.
My suggestion would be the following (this assumes you can train the method on 120 training samples):
- Split into two parts: 120 training and 120 testing.
- Estimate performance measures for all 240 images in the set. (this will require two runs if we split 50/50).
- To test if the real difference between algorithm performance is significanyly different from 0 - perform a paired t-test on the images.
This approach will overcome the limitation of only testing on 30 samples and only using one train/test split. Also it would be a lot faster than doing 30 training/testing splits. And if 120 is too small of a sample size to estimate the model - you can instead split into 3 or more parts.
And in order to me more convincing - you can report the results of t-tests separately for each split instead of combining them into one.