I am using SVM in Matlab (fitcsvm function) to train a classifier for a problem with two classes. Further, I have three features, e.g. A1, A2 and A3, available for each observation composing my full dataset. Initially, my idea was to evaluate different scenarios, such as:

SVM, linear kernel (with all features, with only one feature or combination of two of them); SVM, gaussian kernel involving the same scenarios as for linear kernel. For that, I created a routine where I defined random training and testing sets (75%-25%), n different permutations to be more specific. For each permutation I created two .txt files containing a list of the training and test observations. I did that in order to be able to use the same training and test sets in different scenarios. In general, I see this methodology as a random sub-sampling (or a Monte Carlo) approach to evaluate the sensitivity of the SVM algorithm to the dataset in general. Since I can run training/test sessions with control under the observation composing the groups, I can evaluate the classifiers using a metric, e.g. accuracy (classes are balanced), and check how is the overall performance of each scenario under the same n training/test sets. In the end, I could decide what scenario has the best average performance and I could pick the median classifier out of the n available to be the final classifier.

Does this methodology sound reasonable from a machine learning perspective?

I described it to some colleagues more experienced in ML and they told me I should do k-fold cross-validation instead, since the dataset is not huge. However, I don't understand why k-fold CV would be better than what I did.

Before writing this question I came across the following threads, all helpful, but I could not reach a conclusion about my case:
K-fold vs. Monte Carlo cross-validation
Resampling / simulation methods: monte carlo, bootstrapping, jackknifing, cross-validation, randomization tests, and permutation tests
How many times should we repeat a K-fold CV?
Differences between cross validation and bootstrapping to estimate the prediction error


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.