# Surprising disparity between Confusion matrix values and AUC?

I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix running pretty quick, and after a decent amount of effort I got the PR Curve (with a lot of help from https://stackoverflow.com/questions/29656550/how-to-plot-pr-curve-over-10-folds-of-cross-validation-in-scikit-learn)

The Algorithm consists of doing KFoldStratified, balancing across the 6 classes present, and doing Leave-one-out cross validation. My test set has a single example of each label in the X and y that gets fed in. The Confusion Matrix is generated based on that, and then I use clf.predict(X_test) to generate y probabilities. I separate them into independent lists per label, then I use 'precision_recall_curve' to calculate precision and recall on the combined list per class, then the list containing everything.

Below is the Confusion Matrix and PR Curves I've generated. I don't understand how class 2, for instance, has a seeming perfect classification on the confusion matrix while having a near 0.5 AUC. I'm definitely only using the testing data to calculate both. Any ideas?  • Just to be clear the mean AUC-PR shown is ~77%. And the baseline for it seems to be <20%; AUC-ROC is indeed "useless" close to 50% but here we are shown AUC-PR value that is not bad a priori. What is far more worrying is the huge variance in performance between folds. This suggests to me that the sample used might be too small, the classifier trained too unstable and/or the validation process used not fit for purpose. What is the underlying sample size? Have you considered bootstrap instead of 5-fold cross-validation? Apr 10 at 20:57
• Ah ok so I'm interpreting it wrong due to thinking of it in AUC-ROC terms. Those aren't between folds, those are between classes. There is no between fold data displayed. I have 6 classes with 8 samples of each, and I'm doing Leave one out cross-validation. Sample size is 48, so the training set is 42 and the test set is 6 (one for each label). Can you elaborate on bootstrap? I was under the impression that would land in the same region. Apr 10 at 21:37
• OK, this is a very small sample and building a classifier around it is very challenging. I would strongly suggest getting more data. Apr 10 at 22:56
• Unfortunately, not really possible due to the nature of the data (science lab, whole animal brains, not trivial to get more). So I'm doing the best I can with the data I have. Apr 10 at 23:21
• Maybe then there should be a realignment of the research questions. In that regard: Do you have references to share where other people have managed to answer similar research questions with similar sample sizes? Apr 11 at 1:35