Data Visualization after k-fold Cross Validation step

Question

I need suggestion about data visualisation for cross validated data.

I want to plot data after cross validation and need suggestions how to do that?

I am thinking to plot like this

If I use a classifier e.g SVM, I can plot the results before cross validation step and can show train and test data and misclassifications as in shown in figure.

K-fold Cross validation will do this step (Train/Test split K times with different random split)

In figure, white data points are training data, blue data points are testing data and read data point is misclassification.

I was thinking I can show this plot and show the mean of k-fold cross validation score. But please suggest me any ideas How to visualise data after cross validation step or How to better visualise it.

Purpose of Visualisation: I have intentions to use two classifiers (eg. SVM and logistic regression) and check which classifier performs well and then choose the best performing model for further classification of out of sample data.

Answer 1

Displaying as you go can get quite expensive (time complexity). For classifier selection: Look at Wolpert & McReady's No Free Lunch Theorem, which states that "in the universe of all cost functions, there is no one best classifier." Taking this approach, you also won't know anything about the diversity of your classifiers (you might explore Kuncheva et al). Also, you might explore using "ensemble classifier fusion," and pool the results of both classifiers. Logistic gets very expensive if you have e.g. 7-10 classes, because there is a $p \times p$ variance-covariance for each of the class-1 classes. If you have 20 features and 10 classes then you will have a 190x190 variance covariance matrix that needs to be calculated during every fold of the CV. If you had 1000 objects for such as logistic problem, your code would appear to hang, why, because the 190x190 Hessian matrix will be taking up all the calculations during each fold. If you ran leave-one out CV for this logistic case, you might finish your run in a couple hours. Logistic is a great classifier, but you need to know when it works best-->small class number and small feature number. SVM's on the other hand are "object hungry" and need a lot of objects to successfully train.

Might I recommend starting with kNN, NBC, and multivariate linear regression (LREG) with $\mathbf{y}_i$ set to $\{-1,-1,+1,-1 \} $ for an object that's in class 3 of a 4-class problem, and then assign the object to the class with the most positive $\hat{y}$ out of the 4 $y$'s.

Your approach starts with very expensive choices (SVM, logistic) and might be faster if you started with kNN, NBC, LREG, and then linear discriminant analysis.