Paired Sample T-Test to compare two models over time?

Question

I'm trying to compare some metrics from two models. I'm interested in how well the models do over time, so each row of my data represents the model's performance during a specific time interval. The value of each row is also the average of a 10-fold cross-validation.

Here is a sample of how the data looks:

| model A     | model B     |
|-------------|-------------|
| 0.46109715  | 0.400713107 |
| 0.428206635 | 0.385013369 |
| 0.413500099 | 0.371968505 |
| 0.388656859 | 0.350340418 |
| 0.366748184 | 0.333925309 |
| 0.33125258  | 0.318105248 |
| 0.314924722 | 0.306340307 |
| 0.284139727 | 0.285236266 |
| 0.256875613 | 0.272073157 |

Because the columns are matched in the sense that they correspond to the same time interval, I was using a paired t-test to compare the two models. I'm just not 100% certain this is the right test. And the results of these t-tests are all coming out significant (p way less than .001).

Any help is appreciated!

EDIT: Another approach I thought of is using a repeated measures ANOVA.

Answer 1

I would say that usage of paired t-test and cross validation would be okay but in the way described in paper and article:

1. split data into K-folde
2. train both models for each split
3. get K pairs of metrics (for model A and B)
4. use obtained pairs to perform paired t-test

I hope this link will be helpful - it is about comparing classification algorithm but uses K-fold cv and paired t-test comparing classification algorithms and source paper itself

I don't get one thing, what do you mean by

The value of each row is also the average of a 10-fold cross-validation.

Do you have 9 separate time periods (number of rows) and you run 10-fold cross validation on each of them?