3
$\begingroup$

Are accuracy, AUROC and related measures (sensitivity, specificity) that are commonly used to estimate the performance of a supervised machine learning model using cross-validation etc. collapsible metrics? What I mean by that is suppose we want to estimate the accuracy of an ML classifier for some population composed of 100 individuals, 40 of which are men and 60 are women, and we can find that the accuracy of the classifier is 50% in men and 90% in women, will the test's accuracy in the population of 100 individuals be 50% * 0.4 + 90% * 0.60 = 74%? Or is this not the case?

$\endgroup$

1 Answer 1

1
$\begingroup$

Suddenly the door opens... (Diabolical laughter)
- Nobody expects the Simpson's Paradox!

You are asking if the combined metric can be calculated as a weighted average of the metrics within groups, but it's not that simple. As Simpson's paradox shows us, it is possible for the in-group metrics to be completely different from the total metric. In simple scenarios there are possible corrections, but there is more to this than just taking the weighted average.

So the general answer is "no", you should not expect that you could simply combine the metrics within groups and get the correct total. Also not for all the metrics averaging them makes sense, e.g. weighted average or mean squared errors does make sense, but for RMSE (square root of MSE) it doesn't, as you would need to square the RMSEs first and average the MSEs.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.