I am trying to construct a rule-based classifier on a dataset with 332 instances and 14 features. I am just confused how can I validate the classification model? 10-fold cross validation or holdout method should be used?
Can I just apply the 10-fold cross validation for validation or the model has to be tested by a different set?
Holdout is essentially a 2-fold cross validation. If you perform k-fold cross validation correctly, no extra holdout set is necessary.
Make sure that your predictors are chosen based on the test sets (and not in advance on all the samples). You may also want to think about stratification if appropriate.
There are advantages and disadvantages to both methods of model evaluation. While it is clear to see the advantage of cross-validation in small data (you can use a large amount of your data to train the models and still have all of the data available for evaluation) it can cause problems in later evaluation. First off, your results now come from N different models so the errors can't easily be compared (if your model ends up being highly effected by the training set this can produce wildly different accuracies in different folds). Secondly, while the results come from different models, those models are not independent as (N-2)/(N-1) of the training data were the same between any two pairs of models. Most statistical tests require an assumption of independence so that can lead to difficulty selection a metric for evaluation.
In general, I would go with the hold out method, but if data is really limiting, it may be worth doing cross-validation.
