As a project, i should perform feature selection on small unbalanced datasets with at most 30 features and also at most 300 samples with the help of Genetic algorithm. So, the chromosomes in GA are binary. Also, for evaluating each chromosome, i have used KNN classifier with fixed k=1. I am confused with the usage of training, validation and testing datasets. Could anyone make clear the following questions?

  1. As I have no hyper-parameter (K is fixed for KNN), what is the validation set used for?
  2. Is it reasonable to just split the dataset into just 2 distinct training and test sets and learning the model on the training set with the help of 10-fold CV and at last report the result for test set?
  3. Because of the small number of data samples and features, how could I avoid the overfitting problem? (this is really my main concern)
  4. And at last, suppose that I split the whole data into 3 distinct train, validation and test datasets. Then I could build my classifier on the training data, select the best chromosome according to the validation set, and when the procedure of GA ended report the performance of the best chromosome on the test dataset. At first is this procedure correct? if yes, when reporting the final resulton the test set, according to which dataset I must build my classifier? I mean crossvalidation on test set or building the classifier on the train set and evaluating on the test set. Thanks in advance for your really valuable answers.

2 Answers 2

  1. The purpose of the validation set is probably for computing the cost function optimized by the GA (using the training set error will result in huge over-fitting).

  2. Yes, that would be a better approach. Personally I would use something like nested cross-validation, so the test performance estimate wasn't so dependent on the partitioning into training and test sets (as the dataset is small).

  3. The best way to avoid over-fitting is to do as little tuning of the model as you can. For this reason I would strongly advise against using a GA as a GA is very aggressively optimizing the cost function by evaluating lots of feature combinations. This is especially a problem as k is fixed and k to some extent acts as a regularisation parameter that might mitigate against this to some extent. I would use an algorithm such as relieff to perform feature ranking and then just use the validation set to choose the number of features.


You can use only two splits, use train data for building your model with cross validation. For instance in the case of KNN with 10-fold crossV you evaluate your solutions 10 times, one for each fold, and finally you might consider the average error. This is really more time consuming though. As such, you may benefit from proportionate data sampling from each class of data for evaluation set. So you are to take benefit of both better time complexity and take advantages of contributing all classes for building your model.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.