0
$\begingroup$

I have a small dataset of 87 images. I have done 5-fold cross-validation, training and validating two models on the same data splits. Thus, for each image I have an accuracy score (Dice score). It seems like the paired t-test is the standard method to compare models with this setup, but there's two things I'm confused about:

  1. Isn't the assumption that the values are independent violated, since the models are trained on overlapping datasets?
  2. If the paired t-test can be used: Would the test compare the Dice means (n=5 for each model) or the Dice scores for each image (n=87 for each model)? I've tested both and using the means I get a p-value of 0.06 (so not significant), while using the individual Dice scores it becomes significant. Is one more reliable that the other?
Improve this question
$\endgroup$

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.