1
$\begingroup$

I'm trying to compare some metrics from two models. I'm interested in how well the models do over time, so each row of my data represents the model's performance during a specific time interval. The value of each row is also the average of a 10-fold cross-validation.

Here is a sample of how the data looks:

| model A     | model B     |
|-------------|-------------|
| 0.46109715  | 0.400713107 |
| 0.428206635 | 0.385013369 |
| 0.413500099 | 0.371968505 |
| 0.388656859 | 0.350340418 |
| 0.366748184 | 0.333925309 |
| 0.33125258  | 0.318105248 |
| 0.314924722 | 0.306340307 |
| 0.284139727 | 0.285236266 |
| 0.256875613 | 0.272073157 |

Because the columns are matched in the sense that they correspond to the same time interval, I was using a paired t-test to compare the two models. I'm just not 100% certain this is the right test. And the results of these t-tests are all coming out significant (p way less than .001).

Any help is appreciated!

EDIT: Another approach I thought of is using a repeated measures ANOVA.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.