I have a small dataset of 87 images. I have done 5-fold cross-validation, training and validating two models on the same data splits. Thus, for each image I have an accuracy score (Dice score). It seems like the paired t-test is the standard method to compare models with this setup, but there's two things I'm confused about:
- Isn't the assumption that the values are independent violated, since the models are trained on overlapping datasets?
- If the paired t-test can be used: Would the test compare the Dice means (n=5 for each model) or the Dice scores for each image (n=87 for each model)? I've tested both and using the means I get a p-value of 0.06 (so not significant), while using the individual Dice scores it becomes significant. Is one more reliable that the other?