2
$\begingroup$

I am evaluating a Neural Network performance using 4-fold cross validation. I produced the ROC curve in the picture.

enter image description here

1) How can I average the folds to obtain just one ROC curve?

2) How can I justify this plot to retain it? [ maybe it's useful to plot different folds to see the spread and how stable the model is, could you please explain this? ]

Improve this question
$\endgroup$
7
  • $\begingroup$ What do you want to ultimately show? See Frank's answer here too. $\endgroup$
    – usεr11852
    Commented Mar 12, 2017 at 21:29
  • $\begingroup$ First, I'd like to show just one curve, averaged across all folds. Then, retain this plot, if it's meaningful. $\endgroup$
    – Karl Alexius
    Commented Mar 12, 2017 at 21:39
  • $\begingroup$ Do you get probabilities or just hard labels from each of your folds? $\endgroup$
    – usεr11852
    Commented Mar 12, 2017 at 21:46
  • $\begingroup$ I get probabilities, then I threshold form 0 to 1 with increment 0.001 to obtain the ROC curves. $\endgroup$
    – Karl Alexius
    Commented Mar 12, 2017 at 21:50
  • $\begingroup$ Thresholding them seems a bit arbitrary but it shouldn't be a huge influence. Coming back to what I asked originally: What exactly are you trying to ultimately show? Why do you want to average the curves? Are you looking for a single number (say "average AUC" - which would be wrong) that encapsulate the performance of your classifier? $\endgroup$
    – usεr11852
    Commented Mar 13, 2017 at 0:05

1 Answer 1

Reset to default
1
$\begingroup$

1) Calculate sensitivity and specificity at the incremental thresholds between 0 to 1 for all the folds. Averaging those should give you your desired average ROC Curve.

2)Displaying multiple plots can show you the spread but do not forget that randomly shuffling the same data can result in different spreads as well. The main reason to use cross-validation is mainly motivated by the fact that data spread is random. So try to use cross-validation as a validation tool for your model.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.