Analogue of critical difference (CD) diagram for the comparison over single dataset Does anybody know any fancy way to present the performance of multiple classifiers (in terms of AUC, for example) over just 1, single dataset?
I am familiar with the critical difference diagram for the comparison over multiple datasets, based on post-hoc test performed after the Friedman test (paper from Demsar). However what should one do when the comparison is over a single dataset?
For example, I split my data into training and test subsets, trained multiple classifiers on the training set and then want to present their performance on the test set. How could this be done? If there are good packages for that, I prefer R.
 A: You could compare the (mean) performance of each classifier over individual samples in the test set.  
To check if differences in performances between classifiers are statistically significant, you can compute confidence intervals for the (mean) performances based on standard errors from the individual predictions and then check whether confidence intervals for different classifiers overlap or not. 
Note that some but not all performance measures are means of point-wise measures, i.e. means over measures of individual data points in the test set, e.g. mean squared error which is the mean of squared losses for each data point. Others are aggregate measures (e.g. precision). 
For confidence intervals, you need standard errors of the performance measure. For  mean measures based on point-wise measures, you can simply compute the sample standard error of the mean. For aggregate measures is less straightforward, as you don't have individual data points, but you could still get standard errors by jackknifing or bootstrapping, e.g. delete-one jackknife over the test set. 
Other ways can be found here: T. G. Dietterich. Approximate statistical tests for comparing supervised classification learning al- gorithms. Neural Computation, 10:1895–1924, 1998.
