I'm new in R. I have an original dataset with 25771 variables and 118 samples. I already performed feature selection and split the dataset into 70 30 so i have 82 samples in my training data and 36 in my testing data. Then, I performed SMOTE resampling on my training data only and get a new dataset with 25771 features and 144 samples. By using the output of top features from feature selection result with varying number of features such as 10,20,30 until 100 have been undergoing the classifier package to perform SVM. I already performed SVM using linear and polynomial kernel and then compute the accuracy. I got 91.57% for SVM Linear using 100 features and 75% for SVM with polynomial kernel. But why it doesn't work with RBF kernel? I only get 20% of accuracy using RBF kernel. What should I do next to improve the accuracy of SVM using RBF kernel?
I think there are two things you should potentially be careful about: one is overfitting and the other is tuning the RBF kernel. More specifically:
The RBF kernel (typically) has one free parameter often called the bandwidth. In order to get good performance, this parameter needs to be tuned. Typically, one tries out a (large) grid of candidate values for this parameter and then chooses the best performing value.
Now what does best performing mean? One possible way of approaching the problem is the following. Given that you very few samples in relation to the number of features, your classifier could be prone to overfitting. While splitting the data into training and testing is a good idea (this often referred to as the "holdout" method), you may additionally need a validation set. In this setting the role of the training set is to deliver a trained classifier; the role of validation set is to estimate the performance of the trained classifier. Now for each value of your grid of candidate bandwidth values, you would:
train the classifier on the training set and then estimate the performance using the validation set.
Once you have done this for all the candidate bandwidth values as specified by the grid, you choose bandwidth value that gave you the best performance (e.g. lowest classification error) on the validation set.
Now you train one last time the classifier on both training and validation set.
You estimate the performance of your trained classifier on the yet untouched test set.
Some people advise that the whole process described above is repeated multiple times, each time with a new random split of the data into training, validation and test set. I have avoided prescribing ratios for these three sets, as I have no feeling of what you could be appropriate in such a setting with extremely few samples.
I refer you to some lecture notes that I personally have found useful: http://research.cs.tamu.edu/prism/lectures/iss/iss_l13.pdf
I additionally refer you to the following guide provided with the excellent software LIBSVM where the tuning of the RBF kernel is discussed under the section 3.1: https://www.csie.ntu.edu.tw/~cjlin/papers/guide/guide.pdf