I have a dataset of 165 positive and 1700 negative. For training I balanced them to 132 positive (80%) and 132 negative (80% of minority class). For testing, I leave the default natural distribution ratios (i.e. 33 (20%) positive and 340 (20%) negative).
I am performing 10-fold inner cross-validation using the libsvm default implementation (i.e. I have no access to that data, except the accuracy output at the end). I only use inner-cross-validation for model selection, i.e. to obtain the best hyper-parameters from a grid search on cost and gamma for the RBF kernel.
For outer cross-validation, I want to also perform 10-fold. For these outer-cv predictions (with probability sores) on 'test sets' (i.e. 10 different test sets, since it is 10-fold cv), what is the best practice method to combine these results to generate a ROC or PR curve?
For having a single value I know that the mean average could be taken. However, I want to actually generate the data for plotting the ROC or PR graphs. I already know how to build ROC or PR graphs from a single prediction list, but how about multiple lists from outer-cross-validation? Should I merge the lists into one long list, should I take the mean average for each fold, or the mean average of each point of each fold?
Please note, I am not asking about specific implementations but how to actually implement it from scratch.