1
$\begingroup$

I am trying to construct a rule-based classifier on a dataset with 332 instances and 14 features. I am just confused how can I validate the classification model? 10-fold cross validation or holdout method should be used?

Can I just apply the 10-fold cross validation for validation or the model has to be tested by a different set?

$\endgroup$
2
$\begingroup$

Holdout is essentially a 2-fold cross validation. If you perform k-fold cross validation correctly, no extra holdout set is necessary.

Make sure that your predictors are chosen based on the test sets (and not in advance on all the samples). You may also want to think about stratification if appropriate.

$\endgroup$
  • $\begingroup$ I usually use "holdout" to refer to a validation set (ie, not cross-validation); my point being that it's important to be very clear on what we mean by these terms. $\endgroup$ – gung - Reinstate Monica Aug 11 '16 at 23:46
1
$\begingroup$

There are advantages and disadvantages to both methods of model evaluation. While it is clear to see the advantage of cross-validation in small data (you can use a large amount of your data to train the models and still have all of the data available for evaluation) it can cause problems in later evaluation. First off, your results now come from N different models so the errors can't easily be compared (if your model ends up being highly effected by the training set this can produce wildly different accuracies in different folds). Secondly, while the results come from different models, those models are not independent as (N-2)/(N-1) of the training data were the same between any two pairs of models. Most statistical tests require an assumption of independence so that can lead to difficulty selection a metric for evaluation.

In general, I would go with the hold out method, but if data is really limiting, it may be worth doing cross-validation.

$\endgroup$
  • $\begingroup$ "your results now come from N different models so the errors can't easily be compared" Not necessarily; you take the average across the models, and use that average score as the overall performance of the model. You can then create different models to find the one with the best average performance. You can then train it using all of the data and use that to generate 'real-world' predictions. $\endgroup$ – BigBadMe Jun 23 '18 at 22:58
  • $\begingroup$ @BigMeat You can either compare 1) same data on different models or 2) same model on different data. In this case you would be comparing different models with different data since the hold out set is different in each fold of cross-validation. That is where the problem comes from. $\endgroup$ – Barker Jun 25 '18 at 14:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.