How to validate k-fold cross validation results for classifiers? I've applied 6 classifiers on my data set using 5-fold CV to calculate average AUC for each model. What method do you suggest me to determine which model has the highest AUC?
I've heard of t-test and Mann Whitney test. But I don't know if using these tests are useful or should I select the best model simply by considering the model with the highest AUC by looking at them?
 A: You seem to have a two-part question: 1) how to determine which model has the highest AUC, and 2) how to select the best model.
For 1, I would say that in practice any of the methods you suggest would be fine. Chances are you're going to end up with a few models that have AUCs that are not statistically distinguishable.
That then leads to 2 - if you can't really determine with certainty which model has the best AUC, how do you choose the "best" model? In part that depends on what you are trying to do with the model. Often in the "real world", model complexity is a consideration, since you may be implementing the model in a business process or have to explain it to a client. In that case, simpler is better (it's easier to implement and explain a logistic regression than a neural net). If the costs of the outcome are really high and/or implementation costs are not a concern, then I'd choose the one with the highest AUC.
A: I assume you got this task, from a person who loves p-value. In his mind, AUC is like an effect-size, and p-value is going to show the level of significance. Yes, both are needed !
You can calculate the AUC; e.g if you use a model, that returns the output along with probability score; you can calculate AUC by varying the threshold from 0 to 1 and drawing the ROC curve; (if you need more details with code let me know).
After you have your ROC curve, you can use Wilcox to see if they are significantly different. (as @user777 mentioned, t-test is inappropriate for this case)
Eventually, keep in mind the risk of over-fitting when you are talking about the best model.
psudocode
cross.validation.table = mat.or.vec(nr = 20, nc = 5) #assume you have 100 observation - you you have randomly split them into 5 
also remember each column, indicate which observation should be tested.
for (cnt in 1:5) # start iterations for the CV
{
 test.id = cross.validation.table[,cnt] #right? you have marked who should be tested
 train.id = setdiff(c(1:100), test.id)
 fit = model(x[train.id,], y[train.id]) # train the model
 y_predicted[test.id] = predict(fit, x[test.id,], y[test.id]) # what are the predicted classes for your test samples
}
pred = prediction(y_predicted, y_true)
perf <- performance(pred,"tpr","fpr")
plot(perf ,xlab="1-specificity",ylab="sensitivity")
my.auc = performance(pred, "auc") # can comes from "ROCR"; it also returns ROC curve
...

