Leave-One-Subject-Out cv method

Question

I would like to use a Leave-One-Subject-Out cv on my datasets (I have dataset including 38, 15, 10 participants, respectively). I don't know the hyperparamenters C and gamma of my SVM so I have to estimate them with a cv grid search approach. Someone could explain me step by step how I can do all these thing? That is, how I have to divide the subjects to estimate the hyper parameters, to train the model and finally to test it? I need to do a SVM for each subject?

Answer 1

Dataset division for CV appears to be done at random in most if not all implementations. A good "plain english" explanation of how CV works in general is given by the second answer here, especially the visual.

The LOOCV case is somewhat special in that there is no randomness in assignment to different folds; each observation will, at some point, constitute a "test set" all of its own, so the modeling results should be extremely stable (unless the modelling technique itself induces additional sampling randomness) under repeated CV partitioning attempts. Try doing several iterations of k-fold CV and LOOCV for the same modeling technique, on the same dataset and you will see what I mean.

As for implementation, you can either use the caret package in R, which implements kernlab's svm functions by default or, as @DeepakML recommends, use e1071 which I have had good experiences with as well, but which requires more hand-coding for the grid search.

To use caret, the process is as follows:

1. Create a training parameter control object by using the trainControl function to store training parameters. You can pass the C and gamma values you wish to evaluate as vectors, and specify "LOOCV" as the resampling method parameter.
2. Train your model using the train function.

See here for a more detailed step-by-step instruction.

Should you wish to use a different svm implementation and do not want to submit yourself to the torture that is trying to get caret to interact with packages other than its default implementations, the high-level process is as follows:

1. Create a table to hold all your potential parameter combinations. In R, i recommend using the expand.grid function.
2. Create a place to hold your test results from the model. I recommend simply adding another column to your parameter combination table, as this will keep everything nice, ordered, and easy to keep track of.
3. Create a nested loop and use your selected technique to train your model on the training partition and then test it out on the test partition. The outer loops cycles through each parameter combination, while the inner loop cycles through each partition index that you have created. You will need to store your model performance results from these operations. You can aggregate them and store them as a single value for all different partition indeces for each parameter, or store a separate value for each index, depending on your objective and resource constraints.
4. Compare the performance of each parameter combination and see which one did best according to your chosen metric.
5. Train your model on the entire dataset using the best parameter combination.

As for your three datasets, if they all have the same variables, you should be able to just row-concatenate them. If they have variables they do not share, you'll need to do some legwork to exclude the noncommon ones, either from your concatenated training set, or from the model, as SVM does not play nice with missing values.

A final word of warning. LOOCV can get computationally expensive very quickly as datasets grow and SVM is fairly intensive in its own right. You seem to have a small dataset, so you might be able to get away with it this time, but I would generally caution against this combination of modeling and sampling techniques. And TBH, my personal experience has been that LOOCV is generally not worth the computational expense; I never really go above 10-fold CV in my own modelling.

Answer 2

I am not clear on the 3 datasets you have mentioned. Are these train, test and validation sets? If you are doing CV for finding your hyper-parameters usually you don't need 3 sets. You can keep a final test set which will give the final accuracy of your model.

Typically Leave One Out CV can be done using any statistical modelling software. If you are using R, the package E1071 can do this for you.

Use TUNE.SVM/BEST.SVM to tune the model, the LEAVE ONE OUT CV can be chosen using TUNE.CONTROL parameter within which you will have to set CROSS parameter equal to NUMBER OF DATA POINTS IN YOUR TRAINING SET (tune.control(cross=nrow(dataPoints)))

Thanks