I have 2 dependant dataset D1 and D2 that I used to train and evaluate a regression model within a nested cross-validation that had L outer loop. For each loop, I evaluated the model with the pearson's correlation, which therefore gave me L pearson's correlation coefficient per dataset.
Then, I evaluated if the 2 models give different results with a paired t-test (i.e are the pearson's correlation differences different from 0 on average ?), but I got unsignificant results whereas I actually know that D2 should give significantly higher results. The problem is that the model resulting from D2 adds more variability to the L pearson's correlation. I believe the variance to be inflated by another factor, but testing that assuption and retraining the model is not an option, mainly because it would take way too much time.
So, is there any method to run a paired t-test and accounting for such a problem ? I have read about boostraping the L pearson's correlation to get parameters estimate, or applying a shrinkage coefficient on them to reduce the variance. But I am not sur if these methods are really apropriate (and how to apply them) or if there are better options ?
Edit: from wzbillings's comment, this is indeed a confidence interval that I really need, but that can take into account that inflated variance problem.
