1
$\begingroup$

I am working on a classification problem where my outcome variable is either "Approved" or "Denied". The % of approvals in my dataset is roughly 60% and the denials make up roughly 30%. I have tried multiple models (random forest, decision tree, neural network, and gradient boosting machine). The highest specificity that I can achieve is with the random forest of 0.69. I also tried to balance the data within the "train" function of the caret package by down sampling, over sampling, SMOTE, and ROSE. I performed the sampling only within the training dataset using cross validation (10 folds). I am pretty new to machine learning, so any advice is appreciated. Unfortunately, I cannot provide the datasets or any code that I have written, so I am just looking for general suggestions relating to unbalanced datasets.

$\endgroup$
7
  • $\begingroup$ ... what are the remaining 10%, after you've considered the 60% approvals and 30% denials? And... this is hardly an unbalanced data set, not that unbalanced data is really a problem anyway: stats.stackexchange.com/questions/283170/… $\endgroup$
    – jbowman
    Jul 23, 2019 at 15:26
  • $\begingroup$ Sorry, it's really 68.731% approvals and 31.269% denials. $\endgroup$
    – user254529
    Jul 23, 2019 at 15:32
  • $\begingroup$ I actually tried running the same models without balancing the dataset and the specificity was worse than balancing. $\endgroup$
    – user254529
    Jul 23, 2019 at 15:33
  • $\begingroup$ How many observations & features do you have in your RF, roughly? $\endgroup$
    – jbowman
    Jul 23, 2019 at 15:34
  • $\begingroup$ 23,348 rows for my training set. 24 features. 7 of those are continuous and the rest are categorical. The largest number of levels for my categorical variables is 28. $\endgroup$
    – user254529
    Jul 23, 2019 at 15:38

1 Answer 1

1
$\begingroup$

The easiest way to do this is to change the classification threshold. As you develop your machine learning knowledge, you will see that you destroy much potentially useful information by having any threshold at all, but for now, just changing the threshold sounds like a reasonable first idea.

When you run, for instance, a neural network, you get predictions on a continuum. The standard continuum in this situation is on the interval $[0,1]$. Many software functions turn these into hard classifications by setting a threshold of $0.5$: above the threshold gets classified as one category, and below the threshold gets classified as the other category.

However, you do not have to use $0.5$ as the threshold. If you want more specificity, so harder to be classified as positive, you might want to up the threshold. Perhaps set the threshold to $0.6$ or $0.8$ to get the specificity you desire.

This will improve your specificity at the expense of sensitivity. The tradeoff can be visualized in receiver-operator characteristic (ROC) curves, such as those implemented by pROC::roc in R software. This function even gives a printout of what sensitivity and specificity are achieved at each threshold. Below, I give a quick demonstration of this, and I discuss in more detail here.

library(pROC)
N <- 25
p <- rbeta(N, 1, 1)
y <- rbinom(N, 1, p)
r <- pROC::roc(y, p)
d <- data.frame(
  threshold = r$thresholds,
  sensitivity = r$sensitivities,
  specificity = r$specificities
)
d

################################################################################
#
# OUTPUT
#
################################################################################

    threshold sensitivity specificity
1        -Inf  1.00000000  0.00000000
2  0.03986668  1.00000000  0.07142857
3  0.05315755  1.00000000  0.14285714
4  0.07842079  1.00000000  0.21428571
5  0.12086679  1.00000000  0.28571429
6  0.14478513  1.00000000  0.35714286
7  0.16003195  1.00000000  0.42857143
8  0.21402721  1.00000000  0.50000000
9  0.26453714  1.00000000  0.57142857
10 0.31080317  1.00000000  0.64285714
11 0.35289509  0.90909091  0.64285714
12 0.37692100  0.90909091  0.71428571
13 0.43799047  0.81818182  0.71428571
14 0.49503947  0.81818182  0.78571429
15 0.54152179  0.81818182  0.85714286
16 0.58273907  0.81818182  0.92857143
17 0.60398583  0.72727273  0.92857143
18 0.63121729  0.63636364  0.92857143
19 0.66352988  0.63636364  1.00000000
20 0.73563750  0.54545455  1.00000000
21 0.84121309  0.45454545  1.00000000
22 0.89278788  0.36363636  1.00000000
23 0.92770504  0.27272727  1.00000000
24 0.96569430  0.18181818  1.00000000
25 0.98375395  0.09090909  1.00000000
26        Inf  0.00000000  1.00000000

(There are critics of ROC curves and even sensitivity and specificity in general, among them being Frank Harrell, whose criticisms of these are worth reading.)

Most "classifiers" actually make predictions on a continuum that are then binned according to some threshold to make categorical predictions. If caret does not allow you to access those continuous predictions, the package is less user-friendly than it first seems. (My guess is that you can get them, however.)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.