I am using SVM in Matlab (fitcsvm function) to train a classifier for a problem with two classes. Further, I have three features, e.g. A1, A2 and A3, available for each observation composing my full dataset. Initially, my idea was to evaluate different scenarios, such as:
SVM, linear kernel (with all features, with only one feature or combination of two of them); SVM, gaussian kernel involving the same scenarios as for linear kernel. For that, I created a routine where I defined random training and testing sets (75%-25%), n different permutations to be more specific. For each permutation I created two .txt files containing a list of the training and test observations. I did that in order to be able to use the same training and test sets in different scenarios. In general, I see this methodology as a random sub-sampling (or a Monte Carlo) approach to evaluate the sensitivity of the SVM algorithm to the dataset in general. Since I can run training/test sessions with control under the observation composing the groups, I can evaluate the classifiers using a metric, e.g. accuracy (classes are balanced), and check how is the overall performance of each scenario under the same n training/test sets. In the end, I could decide what scenario has the best average performance and I could pick the median classifier out of the n available to be the final classifier.
Does this methodology sound reasonable from a machine learning perspective?
I described it to some colleagues more experienced in ML and they told me I should do k-fold cross-validation instead, since the dataset is not huge. However, I don't understand why k-fold CV would be better than what I did.
Before writing this question I came across the following threads, all helpful, but I could not reach a conclusion about my case:
K-fold vs. Monte Carlo cross-validation
Resampling / simulation methods: monte carlo, bootstrapping, jackknifing, cross-validation, randomization tests, and permutation tests
How many times should we repeat a K-fold CV?
Differences between cross validation and bootstrapping to estimate the prediction error