I have about 150 samples 1000 features (ranked by their importance by Relieff score). My question is, what would be the best approach to:
choose the hyper parameters
choose the optimal number of features to use
report the accuracy of my model using SVM and kNN (I don’t intend to choose which one of them is best to use, but rather report their accuracy)
First approach: Cross Validation
Split data 80% training and 20% for final testing
Using training data, perform feature ranking with Relieff score
Using training data, loop over the K number of features (starting from the most to the least important) and hyper parameters, using 10-Fold cross validation (to computer the 10-Fold misclassification rate for each combination)
Choose the best K (number of features) and Hyper parameters values, giving the least misclassification rate
Train my algorithm using the training data and optimal parameters and test on the testing data (the 20% of my initial data, which were not used at all for selecting the parameters)
Second approach: Nested Cross Validation
Split data into 10 folds (External Cross Validation)
Do the same as above (Internal Cross Validation) to choose optimal K number of features, and hyper parameters using 10-fold cross validation.
for each external fold, train using 9/10 of data with best chosen parameters and test using 1/10 of data
report the average accuracy of the 10 external folds
Which one should I choose? Any suggestions?