I have created a machine learning model (let's call it Model B) which I hope is better at classifying disease-states than the currently used model (Model A).
I have a dataset of 600 cases (roughly split equally between disease and control) and I've split this into a Training dataset (n=450) and a Validation dataset (n=150).
[The Validation dataset was completely withheld until the end of Model B's creation, so that Model B could be tested against an unseen dataset, thus testing the Model's ability to generalise]
Case 1 : CV on Training dataset
I created Model B using the Training dataset for feature selection. I then performed a 10-fold CV on Training with 1000 repeats. This involved Training being randomly split into a Train (80%) and Test (20%) datasets. Both Model A and B trained on the Train and tested on the Test, and I obtained results on both Models' sensitivity, specificity, AUC and kappa values. As there were 1000 repeats, and both Models were tested on the same Train/Test splits each time, I believe a paired Student T test is sufficient to determine if there is a statistically significant difference between the metrics I tested in Model A vs Model B.
Case 2 : Testing the model on an unseen dataset
I then ran Model B, which I trained on the whole Training dataset (n=450) on the previously unseen Validation dataset (n=150). I did the same with Model A.
The results of the confusion matrix I obtained are below :
- Model A
prediction case control
case 71 8
control 17 54
truth prediction case control case 82 9 control 6 53
So if Model A has Cohen's kappa of 0.66 and Model B has 0.79, what test can I run to check if there's a statistically significant difference? (and the same question for sensitivity, specificity etc.) Because I only run this test once (running it 1000 times would just result in 1000 of the same confusion matrices) I'm not sure if the T test would work as there's no variance. I only have a single value for each metric (i.e sensitivity = 81% vs 93%; specificity = 87% vs 85%).
1. Am I right in using a paired T test to compare results of the 10-fold CV when I ran both models 1000 times within the Training dataset?
2. How can I test if there's a statistically significant difference between the two models from a single confusion matrix?