1
$\begingroup$

So when you read about cross validation, they mention that you split the data in k-1 folds and then test it against one partion of the data. I am currently doing cross validation on my decision trees using the whole data and then test in on the same data. That is wrong if I am not mistaken? Because I want to assess the precision, recall and accuracy from the confusionMatrix/tables I can get once predict my model on the test data.

Is the correct way to divide your data in training (~80%) (cross validation set) and test set (~20%) to later validate your model? This division result in: 60 % training 20 % validation 20 % testing.

$\endgroup$

1 Answer 1

2
$\begingroup$

I am currently doing cross validation on my decision trees using the whole data and then testing on the same data.

What this is doing is evaluating the performance of your model on your training set. Your training error is not as important as the test error because you want to know how well your data will perform on unseen data. Cross validation is a technique used to estimate how well your model will perform on unseen data.

Is the correct way to divide your data in training (~80%) (cross validation set) and test set (~20%) to later validate your model?

Yes, this is common.

This division result in: 60 % training 20 % validation 20 % testing.

Cross validation involves creating several divisions of the data rather than just one, for example a five fold cv would include five different training sets and five different validation sets from your original training set. You would then train a model on each training set and evaluate on each respective validation set. You then average the resulting validation error from each model to calculate an estimate of the performance of your model on unseen data. There are packages in Python and R that can automate this for you.

These resources may be helpful:

  1. Video
  2. Article
$\endgroup$
1
  • $\begingroup$ Thank you for this very enlightening answer. I'm working in R currently and I have a dataset of 957 instances with 29 attributes. I thought this was a very small dataset so I thought 80 (train/k-fold-validation) and 20 for test could be good. So this should be equivalent to 60 % training 20 % validation 20 % testing. $\endgroup$ Commented May 9, 2017 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.