2
$\begingroup$

I have two sets of features (A and B) and I would like to compare the performance of the same classifier, trained on each of these sets.

The two sets correspond to features taken from the same population. Example: feature set A corresponds to the age and gender of a certain group of individuals and feature set B corresponds to their weight and height. So, in the end, the two feature sets will have the same size, and they will correspond to the same population, although they will represent different things.

What is the most suitable statistical test for this comparison?

The model is the same (apart from the hyperparameters, which I optimize), only the set of features are different.

I would like to perform a Monte-Carlo cross validation procedure (randomly sample the test set without replacement) and then perform a paired t-test, since the features come from the same individuals.

Is this a valid approach?

EDIT:

Would a permutation test be more appropriate?

Improve this question
$\endgroup$
4
  • $\begingroup$ What model do you have in mind? $\endgroup$
    – user2974951
    Commented Dec 16, 2020 at 6:26
  • $\begingroup$ I'm currently using XGBoost $\endgroup$
    – asere
    Commented Dec 16, 2020 at 9:06
  • $\begingroup$ Personally, I wouldn't try to mix the two approaches, I would stick to comparing predictive accuracy measures. But if I really had to introduce "significance", then I would use bootstrapping. $\endgroup$
    – user2974951
    Commented Dec 17, 2020 at 9:11
  • $\begingroup$ Thank you for your reply! How would I use bootstrapping in this case? What I obtain in the end is 50 AUCs values for each model $\endgroup$
    – asere
    Commented Dec 17, 2020 at 9:51

1 Answer 1

Reset to default
1
$\begingroup$

If I were to do this, and I wanted to apply some concepts of statistics such as a confidence interval for my parameter (or some sort of "significance"), I would use a bootstrap approach.

In a loop:

  1. Bootstrap your sample
  2. Obtain a score (preferably a proper scoring rule) for models A and B from some hold-out set (a test set or CV)
  3. Compute the difference of scores A - B and save the result

At the end you will have (hopefully) many scores, which represent the distribution of the differences. From here you can apply common approaches with bootstrap results.

For ex.

  1. choose a level for $\alpha$, say $0.05$
  2. compute the $0.025$ and $0.975$ quantile of the scores and also the mean score
  3. compute the distance from the lower and upper quantiles to the mean
  4. add/substract these distances from your original score to obtain a confidence interval around your original score

In general, if such an interval will contain 0, then you cannot reject the hypothesis, that the two models are significantly different, for the chosen $\alpha$.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If I understood you correctly what you described in steps 1-3 is what I have so far. I take multiple random hold out sets and check the performance of both models on that hold out set. In the end I get two distributions of AUCs for each of the models. I haven't, but I can easily compute the differences. What is confusing me in you next steps is the "original score". What does this refer to? $\endgroup$
    – asere
    Commented Dec 17, 2020 at 14:20
  • 1
    $\begingroup$ @asere When using bootstrap to estimate some sort of variability score of your models or parameters, you would usually obtain the average score on your whole (original) training data, and then use resampling to estimate the variability around this "original" score. In the end, you are interested in how your model performs on the original training data, and not on resamples of it. $\endgroup$
    – user2974951
    Commented Dec 17, 2020 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.