I am comparing two models, I have only one accuracy per model. So I wonder, is it correct to generate for example 10.000 bootstrap samples (accuracies) and use the bootstrap samples to make a t-test for comparing the models?
3
-
$\begingroup$ Could you describe the procedure you have in mind in greater detail? $\endgroup$– TimCommented Jun 3, 2021 at 19:29
-
1$\begingroup$ I want to compare two genetics models. I have a training population of ~30.000 individuals and a test population of ~11.000 individuals. So, I run the models and comparing the predicted values of the test with the original values I have the accuracy for each model. Therefore, I have one accuracy per model. My idea is to compare them with bootstraping taking test individuals 10.000 times (and calculating 10.000 accuracies per model) and use this values to compare the models. Is this procedure correct? $\endgroup$– ana_ggCommented Jun 3, 2021 at 19:39
-
$\begingroup$ @ana_gg with this kind of split-sample validation, you just report the accuracy of both models in the validation set. You do not need a statistically significant difference to declare one model is "better" than the other. $\endgroup$– AdamOCommented Jun 3, 2021 at 20:36
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
No. Tests for hypotheses of mean difference using the bootstrap is covered elsewhere. For instance, here: Computing p-value using bootstrap with R
You can't confuse the bootstrap resample as being an observation. This is because the number of bootstrap re-samples is arbitrary. If the hypothesis you wish to test is that the mean is statistically significantly different between the two groups, you can increase the number of bootstrap samples so that any non-zero difference will render a statistically significant result.
-
$\begingroup$ Thanks for your answer. I thought it could be a possiblity. It makes a lot of sense. Do you have any idea how can I compare them so? $\endgroup$– ana_ggCommented Jun 3, 2021 at 20:20