I am trying to implement the following paper: Learning the Kernel Matrix with Semidefinite Programming using cvx toolbox for matlab. My question is about how exactly I can use this method with cross-validation to generate some measure of average accuracy or "generalizability". My situation is the following:
I have 100 trials of some task from a person. These are all labeled, in that we know what type of trial we gave the person to perform: A or B. I now want to build a classifier to see if we can tell them apart without the labels, in the hopes of generalizing into unlabeled data in the future.
So, I went about implementing the 2-norm soft margin equation in the paper (equation 41) using cvx to learn the kernel matrix to try and learn the ideal kernel matrix. I am unclear how exactly this fits in with cross-validation. Does it make sense to split my data into train/test, then learn the kernel matrix using the technique described and test on my test data, and then shuffle my data again into a new train/test set and re-learn the kernel? How does this speak towards the generalization of this when Im essentially learning a new kernel every time? Imagine I do a 5 fold cross validation and get something like: 50, 75, 85, 95, 100%? What can I conclude from that? That my classifier has 81% accuracy? What about the 50% and 75% trials? What if my trails have some mean, but a HUGE variance? How can I be confident?
Also, which one of the five kernel matrices that I've now learned, do I use on the newly collected unseen data?
