0
$\begingroup$

I am using a linear SVM classifier for binary classification problem in a small dataset (60 samples total, balanced i.e. 30+30). My features are 60 in number. I am using an aggregate of filter methods for ranking my features before running a grid search to simultaneously optimize number of features and the C value for linear SVM (all of this in a nested CV framework).

With respect to grid search, I am varying my number of features from 1:p; for each feature size (p), I am checking 11 values of C. For each CV split, I generate feature ranking, choose the top p values, and for each C value, find the misclassification error.

Specifically,

  1. Take training data from outer CV
  2. For each inner CV loop:
    • split into training and test data
    • generate ranking of features using this inner training data
    • for each feature size, for each value of C, find misclassification error on the inner test data
  3. Find average misclassification error across all inner CV
  4. Choose the values of C and p which gave least average misclassification error
  5. Using these values, report unbiased estimate of performance on test data for outer CV

My questions are:

  1. Does this seem like a valid strategy to do simultanous optimization of number of features and other parameters for ML?

  2. Since my overall dataset is quite small, how do I minimize chances of over-fitting while simultaneously optimizing all parameters? Specifically, since I am choosing the model parameters which give me the least misclassification error, it is quite possible that I end up over-fitting; are there strategies similar to 1 standard error rule that could be applied here?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.