1
$\begingroup$

I am creating a couple of models (RF, SVM, LR) and I want to evaluate all of them on a certain PPV (0.7).

This question and this question helped me write my code:

#Run RF
data_ctrl_null <- trainControl(method="cv", number = 10, classProbs = TRUE, summaryFunction=twoClassSummary, savePredictions=T, sampling=NULL)

rf_model <- train(outcome ~ ., data=htn_df, ntree = 2000, tuneGrid = data.frame(mtry = 69), 
  trControl = data_ctrl_null, method= "rf", preProc=c("center","scale"), metric="AUC",importance=TRUE)

#Create an ROC
myRoc <- roc(predictor = rf_model$pred$affirmatory, response = rf_model$pred$obs, positive = 'affirmatory')

#Find threshold for PPV=0.7
coordinates <- coords(myRoc, x = "all", input = "threshold", ret = c("threshold", "ppv"))
plot(t(coordinates))

But the problem there is no threshold the allows me to get a PPV =0.7; the lowest my PPV goes is 0.7417722 at threshold=Inf and the next lowest PPV is 0.7436548 at threshold= 0.7915000. Below is my plot.

Can someone explain to me conceptually what is going on?

enter image description here

EDIT

From user Calimo's explanation below, I determined that the issue is because of the way my code is interpreting positive and negative cases (should be positive=affirmatory and negative=negatory, or control), which is not what I'm seeing.

myRoc    
Call:
roc.default(response = rf_model$pred$obs, predictor = 
   rf_model$pred$affirmatory)

  Data: rf_model$pred$affirmatory in 102 controls (rf_model$pred$obs 
  affirmatory) > 293 cases (rf_model$pred$obs negatory).
  Area under the curve: 0.9008

Saw that the pROC package actually allows you to set cases and controls. I had to modify myRoc:

myRoc_new <- roc(controls = 
 rf_model$pred$affirmatory[rf_model$pred$obs=="negatory"], 
 cases=rf_model$pred$affirmatory[rf_model$pred$obs=="affirmatory"])

Now I can see that my model supports a PPV from 0.25-1 at various thresholds. However, the coordinates only provides me with near thresholds i.e. threshold= 0.4277500 gives PPV= 0.6981132 and threshold= 0.4312500 gives PPV= 0.7047619. This is so close to PPV=0.7 but can I get the exact threshold?

$\endgroup$
4
  • $\begingroup$ Note that the positive = 'affirmatory' argument to roc will be ignored unless I'm very mistaken. $\endgroup$
    – Calimo
    May 13, 2020 at 7:06
  • $\begingroup$ Good to know, I just added that because I've seen it for confusion matrices and wanted the roc to note what was positive $\endgroup$
    – PleaseHelp
    May 13, 2020 at 17:05
  • $\begingroup$ that's a different question really, so you should ask it separately. Stack Overflow would probably be a better fit for that one... $\endgroup$
    – Calimo
    May 14, 2020 at 6:45
  • $\begingroup$ Sure, I can do that. Thanks for your help with the original question! $\endgroup$
    – PleaseHelp
    May 14, 2020 at 16:02

1 Answer 1

2
$\begingroup$

So first I assume that if you get the lowest PPV with threshold=Inf, your classifier assigns lower values to positive instances.

Look at the formula for the positive predictive value:

$\mathrm{PPV} = \frac {\mathrm{TP}} {\mathrm{TP} + \mathrm{FP}}$

When threshold=Inf, you classify all your values as positive. So $\mathrm{TP} + \mathrm{FP}$ = the total sample size, and $\mathrm{TP}$ = the total number of positive, or:

$\mathrm{PPV} = \frac {\mathrm{Positives}} {\mathrm{Total}}$

which is basically the fraction of actual positives in the dataset. The PPV can never go lower than that.

$\endgroup$
1
  • $\begingroup$ Thanks for the robust explanation, that made a lot of sense! I realized that I was experiencing the issue of my controls and cases being switched , so while my expected PPV at threshold=Inf should have been 0.258, my code was giving me a lowest possible PPV=0.742. I rewrote my roc code with cases/controls instead of predictor/response and now I'm seeing PPVs in the right range. However, can I find the exact threshold for a PPV=0.7 (see edited question)? $\endgroup$
    – PleaseHelp
    May 13, 2020 at 23:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.