Confused about Cross validation

Question

So when you read about cross validation, they mention that you split the data in k-1 folds and then test it against one partion of the data. I am currently doing cross validation on my decision trees using the whole data and then test in on the same data. That is wrong if I am not mistaken? Because I want to assess the precision, recall and accuracy from the confusionMatrix/tables I can get once predict my model on the test data.

Is the correct way to divide your data in training (~80%) (cross validation set) and test set (~20%) to later validate your model? This division result in: 60 % training 20 % validation 20 % testing.

Answer 1

I am currently doing cross validation on my decision trees using the whole data and then testing on the same data.

What this is doing is evaluating the performance of your model on your training set. Your training error is not as important as the test error because you want to know how well your data will perform on unseen data. Cross validation is a technique used to estimate how well your model will perform on unseen data.

Is the correct way to divide your data in training (~80%) (cross validation set) and test set (~20%) to later validate your model?

Yes, this is common.

This division result in: 60 % training 20 % validation 20 % testing.

Cross validation involves creating several divisions of the data rather than just one, for example a five fold cv would include five different training sets and five different validation sets from your original training set. You would then train a model on each training set and evaluate on each respective validation set. You then average the resulting validation error from each model to calculate an estimate of the performance of your model on unseen data. There are packages in Python and R that can automate this for you.

These resources may be helpful:

1. Video
2. Article