Say I'm trying to classify a medical condition.

Theres only two classes: Sick and Healthy.

I build a model and I can't split the data because I don't want data from the same patient being in training and test set. So I elect to use Leave-One-Subject-Out, training the model on all subject except one and testing on the left out subject.

So for each test set I have one subject and they are either healthy or sick. So the confusion matrix only contains one class where precision is technically one every time and recall equals accuracy.

I've been reading some papers that claim to use leave-subject-out training and test splits for tasks where patients either have a medical condition or do not. I've seen papers that report accuracy, recall, and precision but I don't understand how you could have precision be less than one if each subject only contains one class. I doubt these papers are lying because I've seen this more than once.

I just want to know whats going on here for them to get precision values that are less than one. Are they doing some kind of averaging or am I missing something and thinking about this in the wrong way? None of the papers explain this either.

I realized I posted on data science instead of here but I think this place would be better to post.


I got my answer in the data science community so I figured I'd share it here for visibility as well. Thanks to user @Erwan Answer link

To summarize it I'll say the following:

You can try pooling: "pool" together all the predictions from each round of testing, and compute the metrics once over these pooled predictions

Basically you could just keep a list of the predictions and true values for each test set. Then after testing for each patient, combine all sets of predictions and true values together and produce one confusion matrix at the end containing the combined predictions and true values from all patients.

Then you'll be able to calculate accuracy, recall, and precision


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.