0
$\begingroup$

I want to predict labels via naive bayes and cross validation and measure the test accuracy. I do understand the principle of cross validation but not completely how to apply it.

My question: Do I have to train and test the model on the whole dataset or do I have to split in test and training set although I use cross-validation?

E.g. either this

train_control <- trainControl(method="cv",savePred=TRUE)
# Use the first 2000 samples to train
naiveModel <- train(as.factor(label)~variable1+variable2+variable3,data[1:2000,], trControl=train_control, method="nb")
# Use the last 400 samples to test
naivePrediction <- predict(naiveModel, data[2000:2400,])
postResample(naivePrediction, as.factor(data[2000:2400,4]))

or that

train_control <- trainControl(method="cv",savePred=TRUE)
# Use the whole dataset to train
naiveModel <- train(as.factor(label)~variable1+variable2+variable3,data, trControl=train_control, method="nb")
# Use the whole dataset to test
naivePrediction <- predict(naiveModel, data)
postResample(naivePrediction, as.factor(data[,4]))
$\endgroup$

2 Answers 2

1
$\begingroup$

You typically implement cross-validation to avoid splitting your data set which would reduce the number of observations used in your training set. Thus, this technique is very powerful when you are working with a limited data set relative to the number of variables in your system.

However, it should be noted that it does not provide as strong of an argument as would a true validation set. If you have a large amount of data compared to your variables, reserving a validation set will give you more confidence that your model adequately captures the feature of interest against the noise.

All of these are tools to understand the generality and performance of your final model and how it will perform when faced with an unknown sample. When your data is limited, use cross-validation (your second code snippet). When your data set is large, cross-validate, but also reserve a true validation set to have more confidence in your model (your first).

$\endgroup$
2
  • $\begingroup$ Good answer. But why can you measure your test accuracy based on a dataset that was used to train? Does it not get overfitted because the algorithm can "remember" all the values? $\endgroup$
    – Kewitschka
    Jan 19, 2016 at 19:47
  • $\begingroup$ So think of the cross-validation as N independent training and validation tests. During any iteration, part of your data is going to train your model, and part is going to validate your model. The data that you reserve to validate does not train the model during this iteration, therefore it is an estimate of your out-of-sample error. You repeat this process and what you will have at the end of the day is N estimates for your out-of-sample error. Then you can take an average of these to get an idea of your performance and the standard deviation to see how much the model will vary. $\endgroup$ Jan 19, 2016 at 23:08
1
$\begingroup$

If you are doing cross-validation on a small dataset. I believe it is acceptable to use the entire dataset to get more accurate predictions. It allows the use of more samples. In Applied Predictive Modeling - Max Kuhn, Kjell Johnson it suggests repeated 10-fold cross-validation for small sample sizes.

On a larger dataset keeping a validation set and doing cross-validation can be useful for getting an even better estimate of out-of-sample error. However, I would not go as far as to say it is required.

The best approach will likely depend on a case-by-case basis.

There are probably different camps of thought when it comes to this, but these are the ones I tend to follow.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.