1
$\begingroup$

I have an imbalanced data set containing 12% of the positive class 88% negative. First, I ran a logistic regression with all my coefficients and got an average accuracy of 0.91 (I know that's not quite good given my class distribution), average sensitivity of 0.34 and average specificity of 0.97. Then I ran an additional logistic regression only using a subset of the coefficients. On average, I got higher accuracy, that is 0.98, lower sensitivity 0.32 and higher specificity, i.e. 0.98 . Is this quite normal or an error in my code? Or is it because of the class distribution, that the classifier using more coefficients is even better in predicting the majority class but worse in predicting the minority class?

$\endgroup$
1
$\begingroup$

This is a classic example of the harm caused by the use of a discontinuous improper scoring rule. A simpler example is shown in the Information Loss chapter in Biostatistics for Biomedical Research, available from http://biostat.mc.vanderbilt.edu/ClinStat. Not only can classification "accuracy" do this it can sometimes tell you that using predictors is worse than just using the overall average risk to predict every subject's outcome. Sensitivity and specificity are also improper accuracy scores and are discussed at length in the Diagnosis chapter of the same notes.

Logistic regression was never intended to be used for the crude and arbitrary task of classification. It is a direct probability model.

$\endgroup$
4
  • $\begingroup$ Thank you for the input - I am definitely going to read the paper. But basically, this output is quite common and you don't think it is an error? $\endgroup$ – Patrick Balada Dec 17 '15 at 13:42
  • $\begingroup$ The only error is likely to be the choice of discontinuous arbitrary accuracy scores. $\endgroup$ – Frank Harrell Dec 17 '15 at 17:38
  • $\begingroup$ @FrankHarrell: Would the paradox (less predictor more predictive) be prevented if one uses the Brier Score? And what exactly does the term direct probability model mean? (a reference would be helpful) Thanks! $\endgroup$ – Clark Chong Dec 18 '15 at 6:05
  • 1
    $\begingroup$ See the wikipedia entry for logistic regression. Given a representative sample the binary logistic model directly estimates Prob($Y=1|X$). Yes if you use the Brier score or a score that comes from the log likelihood itself (logarithmic scoring rule or pseudo $R^2$) you will not see illogical results. $\endgroup$ – Frank Harrell Dec 18 '15 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.