1
$\begingroup$

I have some data examples. If I split the data into three parts and the have some scores for each example of the three parts and then calculate individual AUCs for the three parts

In the next case, I keep all the data examples in one cluster, use the same scores as before and calculate the AUC over this whole dataset. Is it possible that the AUC of the whole bulk could be much greater than the individual parts?

$\endgroup$
0
$\begingroup$

Just to be clear, AUCs are normalized to be a maximum of one (for a perfect classifier). So I think the question is really "can the AUC for all the data together be very different from the average AUC for the three parts?"

This really depends on factors that were not specified in your question:

(a) Was the classifier evaluated by each ROC trained and tested on the same data? If so, then your models will tend to be "tuned" to the test set and the smaller the test set, the more severe this issue will be: the average AUC for each of the three data subsets will tend to be higher than that for the full set of data.

(b) When you split the data, did you split it completely randomly? If so (and if you didn't train on the test set as discussed in (a)) and your data sets are large enough than the average AUC's should be reasonably close. If, OTOH, you split the data in a non-random way, you can get very different results between the two cases. For example, (and I'm making this up) say you were trying to predict which party someone will vote for based on things like education, salary, race, gender, etc. If you split your data by geography so each ROC was based on a particular geographical region you'd likely get more accurate classification than if you threw all regions in together.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.