I have two sets of features (A and B) and I would like to compare the performance of the same classifier, trained on each of these sets.
The two sets correspond to features taken from the same population. Example: feature set A corresponds to the age and gender of a certain group of individuals and feature set B corresponds to their weight and height. So, in the end, the two feature sets will have the same size, and they will correspond to the same population, although they will represent different things.
What is the most suitable statistical test for this comparison?
The model is the same (apart from the hyperparameters, which I optimize), only the set of features are different.
I would like to perform a Monte-Carlo cross validation procedure (randomly sample the test set without replacement) and then perform a paired t-test, since the features come from the same individuals.
Is this a valid approach?
Would a permutation test be more appropriate?