3
$\begingroup$

I've been trying to build a binary classification model using multivariate logistic regression using the caret package in R. My dataset consists of around 20000 observations from which >99% belongs to the X class and only <1% to the Y class, and therefore it is an unbalanced dataset.

According to the book of Max Kuhn and Kjell Johnson (Applied Predictive Modeling, Springer 2013) class imbalance can be managed by either downsampling the majority class or upsampling the minority class of the dataset before training the model. I decided to test both solutions using the same training dataset to compare the results.

The downsampled data set consisted of 822 observations (411 in each class) and the upsampled dataset consisted of 45272 observations (22636 in each class). Both data sets are now "balanced" but I'm not sure which approach to choose. Below I show you the models performances in the training dataset (10-fold CV repeated 5 times).

In terms of sensitivity and specificity, both options (upsampling and downsampling) gave me similar results, although the parameters' standard deviation was 10-fold greater for the downsampled case:

UPSAMPLING results:

ROC                                    
0.7711678          
Sens
0.7011926
Spec
0.697951
ROC SD
0.005977932 
Sens SD
0.00834598
Spec SD
0.009579698

DOWNSAMPLING results:

ROC                               
0.7781663            
Sens
0.7212311
Spec
0.4199943
ROC SD
0.0445779
Sens SD
0.06813285
Spec SD
0.0724861

However, in terms of the significance of the predictors, for the downsampled case only four predictors were significant:

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) -6.561091   1.507289  -4.353 1.34e-05 ***
sexFemale   -0.002311   0.217136  -0.011    0.992    
age          0.044491   0.006457   6.890 5.57e-12 ***
smokingSi    0.004606   0.234458   0.020    0.984    
drinkingSi   0.017497   0.185291   0.094    0.925    
diabHistSi   0.732457   0.163528   4.479 7.50e-06 ***
htDXSi       0.010499   0.222508   0.047    0.962    
height      -0.007022   0.007923  -0.886    0.375    
waist        0.022091   0.005598   3.947 7.93e-05 ***
aveSP        0.024395   0.005420   4.501 6.77e-06 ***

In contrast, for the upsampled case, all of the predictors were significant:

              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -6.0755790  0.2305014 -26.358  < 2e-16 ***
sexFemale   -0.1409143  0.0302186  -4.663 3.11e-06 ***
age          0.0304018  0.0008032  37.849  < 2e-16 ***
smokingSi   -0.0691276  0.0309232  -2.235  0.02539 *  
drinkingSi   0.0538318  0.0243686   2.209  0.02717 *  
diabHistSi   0.6493554  0.0214752  30.238  < 2e-16 ***
htDXSi      -0.0809236  0.0281700  -2.873  0.00407 ** 
height      -0.0105632  0.0012510  -8.444  < 2e-16 ***
waist        0.0312969  0.0008087  38.699  < 2e-16 ***
aveSP        0.0237232  0.0006948  34.146  < 2e-16 ***

Which one you think is better? On the one hand, downsampling the data set I'm neglecting almost 20000 observations belonging to the majority class. On the other hand, when I upsample the minority class I'm duplicating the same 400 observations several times...

I know that I can look for a different classification threshold in the ROC curve instead of using down or upsampling to manage the original unbalanced dataset, but I've tried that and I'm not getting good results.

I also know that other methods like support vector machines can use a cost function in order to identify cases of the minority class, but I need the model to be interpretable and "user friendly". That's why I'm using logistic regression.

$\endgroup$
3
$\begingroup$

NEVER use downsampling to make a method work. If the method is any good it will work under imbalance. Removal of samples is not scientific. Logistic regression works well under extreme imbalance. Also (1) logistic regression is not a classification method, (2) make sure you use proper accuracy scoring rules, and (3) logistic regression is not a multivariate (multiple dependent variables) method. It is a multivariable regression method.

$\endgroup$
  • $\begingroup$ Thank you! I agree with you. I don't like to remove samples (I'm not that comfortable duplicating samples either) but I'm getting 0 sensibility and 1 specificity with an accuracy of 98%. I've been using the metric "Kappa" to determine the best model but I get the same results. Is upsampling or the SMOTE algorithm better? $\endgroup$ – Gerardo Felix Mar 1 '16 at 14:57
  • 1
    $\begingroup$ If you insist on using discontinuous improper accuracy scoring rules you will continue to see such anomalies. $\endgroup$ – Frank Harrell Mar 1 '16 at 16:43
  • 2
    $\begingroup$ What scoring rules should I be using then? Can you elaborate a little bit about it? I'd appreciate it... $\endgroup$ – Gerardo Felix Mar 1 '16 at 22:20
  • 1
    $\begingroup$ See Section 10.6 (Chapter 10) of Course Notes at biostat.mc.vanderbilt.edu/rms $\endgroup$ – Frank Harrell Mar 2 '16 at 2:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.