How to statistically compare the same classifier on two different sets of features? I have two sets of features (A and B) and I would like to compare the performance of the same classifier, trained on each of these sets.
The two sets correspond to features taken from the same population. Example: feature set A corresponds to the age and gender of a certain group of individuals and feature set B corresponds to their weight and height. So, in the end, the two feature sets will have the same size, and they will correspond to the same population, although they will represent different things.
What is the most suitable statistical test for this comparison?
The model is the same (apart from the hyperparameters, which I optimize), only the set of features are different.
I would like to perform a Monte-Carlo cross validation procedure (randomly sample the test set without replacement) and then perform a paired t-test, since the features come from the same individuals.
Is this a valid approach?
EDIT:
Would a permutation test be more appropriate?
 A: If I were to do this, and I wanted to apply some concepts of statistics such as a confidence interval for my parameter (or some sort of "significance"), I would use a bootstrap approach.
In a loop:

*

*Bootstrap your sample

*Obtain a score (preferably a proper scoring rule) for models A and B from some hold-out set (a test set or CV)

*Compute the difference of scores A - B and save the result

At the end you will have (hopefully) many scores, which represent the distribution of the differences. From here you can apply common approaches with bootstrap results.
For ex.


*choose a level for $\alpha$, say $0.05$

*compute the $0.025$ and $0.975$ quantile of the scores and also the mean score

*compute the distance from the lower and upper quantiles to the mean

*add/substract these distances from your original score to obtain a confidence interval around your original score

In general, if such an interval will contain 0, then you cannot reject the hypothesis, that the two models are significantly different, for the chosen $\alpha$.
