5
$\begingroup$

I've been trying to build a binary classification model using multivariate logistic regression using the caret package in R. My dataset consists of around 20000 observations from which >99% belongs to the X class and only <1% to the Y class, and therefore it is an unbalanced dataset.

According to the book of Max Kuhn and Kjell Johnson (Applied Predictive Modeling, Springer 2013) class imbalance can be managed by either downsampling the majority class or upsampling the minority class of the dataset before training the model. I decided to test both solutions using the same training dataset to compare the results.

The downsampled data set consisted of 822 observations (411 in each class) and the upsampled dataset consisted of 45272 observations (22636 in each class). Both data sets are now "balanced" but I'm not sure which approach to choose. Below I show you the models performances in the training dataset (10-fold CV repeated 5 times).

In terms of sensitivity and specificity, both options (upsampling and downsampling) gave me similar results, although the parameters' standard deviation was 10-fold greater for the downsampled case:

UPSAMPLING results:

ROC                                    
0.7711678          
Sens
0.7011926
Spec
0.697951
ROC SD
0.005977932 
Sens SD
0.00834598
Spec SD
0.009579698

DOWNSAMPLING results:

ROC                               
0.7781663            
Sens
0.7212311
Spec
0.4199943
ROC SD
0.0445779
Sens SD
0.06813285
Spec SD
0.0724861

However, in terms of the significance of the predictors, for the downsampled case only four predictors were significant:

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept) -6.561091   1.507289  -4.353 1.34e-05 ***
sexFemale   -0.002311   0.217136  -0.011    0.992    
age          0.044491   0.006457   6.890 5.57e-12 ***
smokingSi    0.004606   0.234458   0.020    0.984    
drinkingSi   0.017497   0.185291   0.094    0.925    
diabHistSi   0.732457   0.163528   4.479 7.50e-06 ***
htDXSi       0.010499   0.222508   0.047    0.962    
height      -0.007022   0.007923  -0.886    0.375    
waist        0.022091   0.005598   3.947 7.93e-05 ***
aveSP        0.024395   0.005420   4.501 6.77e-06 ***

In contrast, for the upsampled case, all of the predictors were significant:

              Estimate Std. Error z value Pr(>|z|)    
(Intercept) -6.0755790  0.2305014 -26.358  < 2e-16 ***
sexFemale   -0.1409143  0.0302186  -4.663 3.11e-06 ***
age          0.0304018  0.0008032  37.849  < 2e-16 ***
smokingSi   -0.0691276  0.0309232  -2.235  0.02539 *  
drinkingSi   0.0538318  0.0243686   2.209  0.02717 *  
diabHistSi   0.6493554  0.0214752  30.238  < 2e-16 ***
htDXSi      -0.0809236  0.0281700  -2.873  0.00407 ** 
height      -0.0105632  0.0012510  -8.444  < 2e-16 ***
waist        0.0312969  0.0008087  38.699  < 2e-16 ***
aveSP        0.0237232  0.0006948  34.146  < 2e-16 ***

Which one you think is better? On the one hand, downsampling the data set I'm neglecting almost 20000 observations belonging to the majority class. On the other hand, when I upsample the minority class I'm duplicating the same 400 observations several times...

I know that I can look for a different classification threshold in the ROC curve instead of using down or upsampling to manage the original unbalanced dataset, but I've tried that and I'm not getting good results.

I also know that other methods like support vector machines can use a cost function in order to identify cases of the minority class, but I need the model to be interpretable and "user friendly". That's why I'm using logistic regression.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
13
$\begingroup$

NEVER use downsampling to make a method work. If the method is any good it will work under imbalance. Removal of samples is not scientific. Logistic regression works well under extreme imbalance. Also (1) logistic regression is not a classification method, (2) make sure you use proper accuracy scoring rules, and (3) logistic regression is not a multivariate (multiple dependent variables) method. It is a multivariable regression method.

Improve this answer
$\endgroup$
16
  • 1
    $\begingroup$ Thank you! I agree with you. I don't like to remove samples (I'm not that comfortable duplicating samples either) but I'm getting 0 sensibility and 1 specificity with an accuracy of 98%. I've been using the metric "Kappa" to determine the best model but I get the same results. Is upsampling or the SMOTE algorithm better? $\endgroup$
    – Gerardo Felix
    Mar 1, 2016 at 14:57
  • 1
    $\begingroup$ If you insist on using discontinuous improper accuracy scoring rules you will continue to see such anomalies. $\endgroup$
    – Frank Harrell
    Mar 1, 2016 at 16:43
  • 3
    $\begingroup$ What scoring rules should I be using then? Can you elaborate a little bit about it? I'd appreciate it... $\endgroup$
    – Gerardo Felix
    Mar 1, 2016 at 22:20
  • 1
    $\begingroup$ See Section 10.6 (Chapter 10) of Course Notes at biostat.mc.vanderbilt.edu/rms $\endgroup$
    – Frank Harrell
    Mar 2, 2016 at 2:39
  • 1
    $\begingroup$ You think that the fact that tech companies do something exceedlingly stupid makes it OK? ANY method that requires you to delete data is ill-advised. And the link I provided above has been updated to hbiostat.org/rms $\endgroup$
    – Frank Harrell
    Jan 19, 2021 at 13:16
0
$\begingroup$

Yes upsampling is better if your machine can handle the bigger dataset to train especially if you're testing for statistically significant associations in your model. Due to the fact that more samples = more statistical power and lower standard error estimate which is why your results are the way they are.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Upsampling does not create information out of thin air so it does not increase statistical power when you use the correct standard errors. Upsampling or downsampling only help if you use an invalid accuracy scoring rule. $\endgroup$
    – Frank Harrell
    Nov 9, 2022 at 12:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.