How to visualize 10-fold cross-validation results?

I want to visually compare the 10-fold cross-validation results of two models. (Note that the data are paired, as both models are fit to the same folds and tested against the same held out fold on each iteration.) First, I though I would use box plots to compare the models, but I think 10 samples for each box plot is too few, isn't it? Another alternative would be to draw means side by side and standard error bars.

What is the best way to compare the 10-fold cross-validation results of two models?

You're right: Boxplots are a good way to show data, but when there are few data it is recommended to just plot the actual data themselves. Further, you are most interested in showing the relationship within the pairings (probably the difference, but possibly their ratio, etc.), not the raw data, although they could be plotted in the background. For a small number of single values, the standard recommendation is to use a dot plot.

Below I quickly create some fake data and show a dot plot (using R):

set.seed(6534)  # this makes the example exactly reproducible
m1   = rbeta(10, 8, 2)
m2   = rbeta(10, 7, 3)
difs = m1 - m2
windows()
dotchart(difs, labels=paste("iteration",1:10), xlab="difference (m1 - m2)")
abline(v=0, col="gray") Of course, what is the 'right' plot depends on what you want to show. Above, I'm assuming you are looking at which model might be better. It would be a lot less common, but it is thinkable that someone might want to show how different models are nonetheless similar. In that case, a Bland-Altman plot (Tukey mean-difference plot) can be used to assess agreement. An example is shown below. It's hard to say too much, but you see smaller (and possibly negative) differences when the mean is higher. That's presumably because you are against the upper bound.

library(BlandAltmanLeh)
bland.altman.plot(m1, m2) 