0
$\begingroup$

I have been doing some training of basic models for a certain binary outcome, and most of the training has been on optimizing the AUC. But when I plot the precision recall curves, I get essentially the exact same value for each model which seems unlikely. This dataset is quite biased, about 5% positive class. attached is the code:

 training = read.csv("trainingdata.csv")
 testing= read.csv("testingdata.csv")
 rftraining=read.csv("RFTraining.csv")
 rftesting=read.csv("RFTesting.csv") 
 svmpreds = read.csv("svmpreds.csv") 
 GBpreds = as.data.frame(read.csv("leepreduntuned.csv")) 
 RFpreds = read.csv("forestpreds.csv") 
 ENPLRpreds = read.csv("enplrpreds.csv")

 Performance <- roc(testing$amputation ~ GBpreds$x + ENPLRpreds$s0 +
 RFpreds$YES+svmpreds$x) #+ nntpreds[,1])
##making AU-ROC curve for the predictions 
 g4 <- ggroc(Performance, aes=c("color"), legacy.axes = TRUE) +
       scale_color_manual(name="Models", labels=c("Gradient Boosted","Net
       Penalized Logit","Random Forest", "Support Vector Machine"),
       values=c("red","green","blue", "black")) +
       xlab("False Positive Rate(1-Specificity)") +
       ylab("True Positive(Sensitivity)")+
       geom_segment(aes(x = 1, xend = 0,y=1, yend=0), color = "grey", linetype = "dashed") 
 g4

auc for all models

GBpr <- pr.curve(scores.class0 = GBpreds$x, scores.class1 = (1-GBpreds$x), curve = T)
plot(GBpr)
RFPRC <- pr.curve(scores.class0 = RFpreds$YES, scores.class1 = RFpreds$NO, curve = T)
plot(RFPRC)
SVMPRC <-  pr.curve(scores.class0 = svmpreds$x, scores.class1 = (1-svmpreds$x), curve = T)
plot(SVMPRC)

example PR curve, they all look like this with exactly the same AUPRC

$\endgroup$

1 Answer 1

1
$\begingroup$

I have solved my problem...In the PRROC commands I was using

    GBpr <- pr.curve(scores.class0 = GBpredsπ‘₯,π‘ π‘π‘œπ‘Ÿπ‘’π‘ .π‘π‘™π‘Žπ‘ π‘ 1=(1βˆ’πΊπ΅π‘π‘Ÿπ‘’π‘‘π‘ x), curve = T) plot(GBpr)

when i should have been using

    GBpr <- pr.curve(scores.class0 = testing_set$amputation, weights.class0= GBPred$x, curve = T) plot(GBpr)

which is proper implementation of the PRROC code.

$\endgroup$

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.