My aim is to find the best C for a linear SVM classification using libsvm, I have 120 instances in total and 2 classes which I want to classify.
I have question regarding partitioning a the dataset into these parts: 1) training for optimisation during parameter C search 2) training for model construction 3) testing (representing future incoming data)
Using cross-validation and grid-search, I'd like to find the optimal C for my dataset. There seem to be different suggestions on how to partition data into these three parts
I could first divide my data into training (50%) and testing (50%). From the training, I use one half (25% of total data) for cross-validation for optimisation of C. Once I found the best C, I use THE OTHER HALF of the training data (25% of total data) , to construct the model using the best C. I then validate the model against the 50% testing data, the resulting accuracy is the final accuracy. This way, neither the data that went in for the model-construction nor the testing data have ever been used during optimisation.
Use 50% for training and 50% for testing. The entire 50% for training is used for cross-validation for optimisation of the parameter C. Once I found the best C, I train on the full 50% of the training data to obtain the model, and validate it on the test data.
I think the advantage of 1. is that neither model construction and testing data were in the optimisation process, hence there is less of an overfitting problem. I read differing accounts online though, so any input would be greatly appreciated. Does the method depend on how much data there is available? In my case, I don't have that many instances, so I am not sure if method 1. is recommended. Reference to other relevant sources/textbooks would also be greatly appreciated.