I'm using a linear SVM to perform classification on a dataset.
It seems to me that there are many ways to visualize my results and report their statistical significance, and I'm unsure of best practices.
Below I will list the two main options I've considered.
Take all the data. Compute a single classification accuracy. (This includes dividing into training and testing sets, doing cross-validation, averaging over n-folds, etc. But the end result is a single accuracy score that corresponds to the full dataset.) Plot this value.
For error bars, plot bootstrapped CIs. (The CIs would have been computed from, say, getting 1,000 accuracy values for 1,000 bootstraps.)
To determine statistical significance, compare the single empirical accuracy to accuracies for 1,000 reshufflings of the labels.
Take 1,000 bootstraps of the data. As my empirical accuracy, take the mean accuracy for these bootstraps.
For error bars, plot the SD of the bootstrapped distribution.
To determine statistical significance, compare the bootstrapped mean to 1,000 means computed from 1,000 reshufflings of each bootstrap (mean across bootstraps for each reshuffling).
Which of these strategies makes the most sense?