1
$\begingroup$

I'm working on getting a read out of a Logistic regression classification model (setup in Python via Scikit-learn's LogisticRegression() wrapped in a OneVsRestClassifier()). I got the confusion matrix running pretty quick, and after a decent amount of effort I got the PR Curve (with a lot of help from https://stackoverflow.com/questions/29656550/how-to-plot-pr-curve-over-10-folds-of-cross-validation-in-scikit-learn)

The Algorithm consists of doing KFoldStratified, balancing across the 6 classes present, and doing Leave-one-out cross validation. My test set has a single example of each label in the X and y that gets fed in. The Confusion Matrix is generated based on that, and then I use clf.predict(X_test) to generate y probabilities. I separate them into independent lists per label, then I use 'precision_recall_curve' to calculate precision and recall on the combined list per class, then the list containing everything.

Below is the Confusion Matrix and PR Curves I've generated. I don't understand how class 2, for instance, has a seeming perfect classification on the confusion matrix while having a near 0.5 AUC. I'm definitely only using the testing data to calculate both. Any ideas?

enter image description here

enter image description here

Improve this question
$\endgroup$
6
  • $\begingroup$ Just to be clear the mean AUC-PR shown is ~77%. And the baseline for it seems to be <20%; AUC-ROC is indeed "useless" close to 50% but here we are shown AUC-PR value that is not bad a priori. What is far more worrying is the huge variance in performance between folds. This suggests to me that the sample used might be too small, the classifier trained too unstable and/or the validation process used not fit for purpose. What is the underlying sample size? Have you considered bootstrap instead of 5-fold cross-validation? $\endgroup$
    – usεr11852
    Commented Apr 10, 2023 at 20:57
  • $\begingroup$ Ah ok so I'm interpreting it wrong due to thinking of it in AUC-ROC terms. Those aren't between folds, those are between classes. There is no between fold data displayed. I have 6 classes with 8 samples of each, and I'm doing Leave one out cross-validation. Sample size is 48, so the training set is 42 and the test set is 6 (one for each label). Can you elaborate on bootstrap? I was under the impression that would land in the same region. $\endgroup$
    – AnimNeuroGrad
    Commented Apr 10, 2023 at 21:37
  • $\begingroup$ OK, this is a very small sample and building a classifier around it is very challenging. I would strongly suggest getting more data. $\endgroup$
    – usεr11852
    Commented Apr 10, 2023 at 22:56
  • $\begingroup$ Unfortunately, not really possible due to the nature of the data (science lab, whole animal brains, not trivial to get more). So I'm doing the best I can with the data I have. $\endgroup$
    – AnimNeuroGrad
    Commented Apr 10, 2023 at 23:21
  • $\begingroup$ Maybe then there should be a realignment of the research questions. In that regard: Do you have references to share where other people have managed to answer similar research questions with similar sample sizes? $\endgroup$
    – usεr11852
    Commented Apr 11, 2023 at 1:35

1 Answer 1

Reset to default
0
$\begingroup$

Precision-recall curves are not ROC curves, they have a different interpretation and they do not have the chance level at 0.5. Also, they are sensitive to class imbalance. You can have high accuracy by a model that for example only predicts the majority class in an imbalanced dataset, but the precision-recall curve will be bad.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Got it. The classes are balanced in the training and testing sets here. I am still unsure of how Class 1 AUC-PR is around 44% while the Confusion matrix shows it is correctly identified 88% of the time (since this is just across 8 samples I have for each of this 6 classes, means it got 1 wrong and 7 correct across the splits). $\endgroup$
    – AnimNeuroGrad
    Commented Apr 10, 2023 at 21:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.