I would like to compare the classifier performance when there is a newly added variable to the dataset. Say the original prediction was with 10 input variables. The new one is with 11 inputs.
My purpose is to check whether it is practically useful to collect this new input variable. I am particularly interested in comparing the specificity performance due to my setting.
What would be the correct way to statistically test whether it is indeed useful in terms of specificity to collect this new input?
What I did was:
I have approximately 300 data points.
- I ran a simple decision tree algorithm with 10-fold cross-validation with stratified sampling for the original dataset and the new dataset. I used the same folds for both.
- Then, conducted an unpaired t-test for the specificities based on the folds for each dataset.
I am unsure about step (2). I am not sure if I should do a paired or an unpaired t-test. Or maybe some other test?
I haven't been able to find a source dealing with this yet. There is a similar question here, but there is no specific answer for which statistical test would be appropriate.