I've been trying to build a binary classification model using multivariate logistic regression using the caret package in R. My dataset consists of around 20000 observations from which >99% belongs to the X class and only <1% to the Y class, and therefore it is an unbalanced dataset.
According to the book of Max Kuhn and Kjell Johnson (Applied Predictive Modeling, Springer 2013) class imbalance can be managed by either downsampling the majority class or upsampling the minority class of the dataset before training the model. I decided to test both solutions using the same training dataset to compare the results.
The downsampled data set consisted of 822 observations (411 in each class) and the upsampled dataset consisted of 45272 observations (22636 in each class). Both data sets are now "balanced" but I'm not sure which approach to choose. Below I show you the models performances in the training dataset (10-fold CV repeated 5 times).
In terms of sensitivity and specificity, both options (upsampling and downsampling) gave me similar results, although the parameters' standard deviation was 10-fold greater for the downsampled case:
UPSAMPLING results:
ROC
0.7711678
Sens
0.7011926
Spec
0.697951
ROC SD
0.005977932
Sens SD
0.00834598
Spec SD
0.009579698
DOWNSAMPLING results:
ROC
0.7781663
Sens
0.7212311
Spec
0.4199943
ROC SD
0.0445779
Sens SD
0.06813285
Spec SD
0.0724861
However, in terms of the significance of the predictors, for the downsampled case only four predictors were significant:
Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -6.561091 1.507289 -4.353 1.34e-05 ***
sexFemale -0.002311 0.217136 -0.011 0.992
age 0.044491 0.006457 6.890 5.57e-12 ***
smokingSi 0.004606 0.234458 0.020 0.984
drinkingSi 0.017497 0.185291 0.094 0.925
diabHistSi 0.732457 0.163528 4.479 7.50e-06 ***
htDXSi 0.010499 0.222508 0.047 0.962
height -0.007022 0.007923 -0.886 0.375
waist 0.022091 0.005598 3.947 7.93e-05 ***
aveSP 0.024395 0.005420 4.501 6.77e-06 ***
In contrast, for the upsampled case, all of the predictors were significant:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -6.0755790 0.2305014 -26.358 < 2e-16 ***
sexFemale -0.1409143 0.0302186 -4.663 3.11e-06 ***
age 0.0304018 0.0008032 37.849 < 2e-16 ***
smokingSi -0.0691276 0.0309232 -2.235 0.02539 *
drinkingSi 0.0538318 0.0243686 2.209 0.02717 *
diabHistSi 0.6493554 0.0214752 30.238 < 2e-16 ***
htDXSi -0.0809236 0.0281700 -2.873 0.00407 **
height -0.0105632 0.0012510 -8.444 < 2e-16 ***
waist 0.0312969 0.0008087 38.699 < 2e-16 ***
aveSP 0.0237232 0.0006948 34.146 < 2e-16 ***
Which one you think is better? On the one hand, downsampling the data set I'm neglecting almost 20000 observations belonging to the majority class. On the other hand, when I upsample the minority class I'm duplicating the same 400 observations several times...
I know that I can look for a different classification threshold in the ROC curve instead of using down or upsampling to manage the original unbalanced dataset, but I've tried that and I'm not getting good results.
I also know that other methods like support vector machines can use a cost function in order to identify cases of the minority class, but I need the model to be interpretable and "user friendly". That's why I'm using logistic regression.