I've been trying to build a binary classification model using multivariate logistic regression using the caret package in R. My dataset consists of around 20000 observations from which >99% belongs to the X class and only <1% to the Y class, and therefore it is an unbalanced dataset.
According to the book of Max Kuhn and Kjell Johnson (Applied Predictive Modeling, Springer 2013) class imbalance can be managed by either downsampling the majority class or upsampling the minority class of the dataset before training the model. I decided to test both solutions using the same training dataset to compare the results.
The downsampled data set consisted of 822 observations (411 in each class) and the upsampled dataset consisted of 45272 observations (22636 in each class). Both data sets are now "balanced" but I'm not sure which approach to choose. Below I show you the models performances in the training dataset (10-fold CV repeated 5 times).
In terms of sensitivity and specificity, both options (upsampling and downsampling) gave me similar results, although the parameters' standard deviation was 10-fold greater for the downsampled case:
ROC 0.7711678 Sens 0.7011926 Spec 0.697951 ROC SD 0.005977932 Sens SD 0.00834598 Spec SD 0.009579698
ROC 0.7781663 Sens 0.7212311 Spec 0.4199943 ROC SD 0.0445779 Sens SD 0.06813285 Spec SD 0.0724861
However, in terms of the significance of the predictors, for the downsampled case only four predictors were significant:
Coefficients: Estimate Std. Error z value Pr(>|z|) (Intercept) -6.561091 1.507289 -4.353 1.34e-05 *** sexFemale -0.002311 0.217136 -0.011 0.992 age 0.044491 0.006457 6.890 5.57e-12 *** smokingSi 0.004606 0.234458 0.020 0.984 drinkingSi 0.017497 0.185291 0.094 0.925 diabHistSi 0.732457 0.163528 4.479 7.50e-06 *** htDXSi 0.010499 0.222508 0.047 0.962 height -0.007022 0.007923 -0.886 0.375 waist 0.022091 0.005598 3.947 7.93e-05 *** aveSP 0.024395 0.005420 4.501 6.77e-06 ***
In contrast, for the upsampled case, all of the predictors were significant:
Estimate Std. Error z value Pr(>|z|) (Intercept) -6.0755790 0.2305014 -26.358 < 2e-16 *** sexFemale -0.1409143 0.0302186 -4.663 3.11e-06 *** age 0.0304018 0.0008032 37.849 < 2e-16 *** smokingSi -0.0691276 0.0309232 -2.235 0.02539 * drinkingSi 0.0538318 0.0243686 2.209 0.02717 * diabHistSi 0.6493554 0.0214752 30.238 < 2e-16 *** htDXSi -0.0809236 0.0281700 -2.873 0.00407 ** height -0.0105632 0.0012510 -8.444 < 2e-16 *** waist 0.0312969 0.0008087 38.699 < 2e-16 *** aveSP 0.0237232 0.0006948 34.146 < 2e-16 ***
Which one you think is better? On the one hand, downsampling the data set I'm neglecting almost 20000 observations belonging to the majority class. On the other hand, when I upsample the minority class I'm duplicating the same 400 observations several times...
I know that I can look for a different classification threshold in the ROC curve instead of using down or upsampling to manage the original unbalanced dataset, but I've tried that and I'm not getting good results.
I also know that other methods like support vector machines can use a cost function in order to identify cases of the minority class, but I need the model to be interpretable and "user friendly". That's why I'm using logistic regression.