How can I perform 10-fold cross validation by manually constructing datasets?

Question

I am working in text classification in RapidMiner where, because of the nature of my problem, I cannot use the built-in k-fold cross validation strategy, so I decided to create 10 copies of my dataset manually in order to incorporate the k-fold cross validation $(k = 10)$. However I have a few confusions regarding it. I have a dataset of 4000 samples distributed in 20 categories.

1. I am thinking to incorporate 10-fold cross validation by creating 10 copies of same dataset where 90% (3600) will be used for training and 10% (400) for testing. In each of the 10 copies of dataset, the test set will be different from other datasets. In the end of the classification task I will average their results. Am I right about my interpretation of 10-fold cross validation?

2. When we say to divide data in k-folds, use k-1 folds for training, and use the last one for testing, do we mean to train the classifier for k-1 times on k-1 folds, or do we mean to train the classifier only once on k-1 folds?

   E.g. we have a data set of 4000 samples which is divided in 10 folds with 400 samples in each fold. By training the classifier on k-1 folds, do we mean train the classifier k-1 times (here 9) on k-1 folds (9) with 400 samples in each fold, or do we simply mean to train the classifier once on (400 × 9 samples) and test it on the last fold?

Answer 1

Cross validation with k folds means you will have to split you data set in k disjoint groups. In your case for 10-folds you split your data set in 10 disjoint groups each with 400 samples ($G_i$ with $i$ from 1 to 10). Usually the groups should have roughly the same size.

Now do the following:

1. Train your classifier on $Train_1 = G_2\cup G_3 \cup ... \cup G_{10}$ and test it on $Test_1 = G_1$. Save test results for later use.
2. Train your classifier on $Train_2 = G_1 \cup G_3 \cup .. \cup G_{10}$ and test on $Test_2 = G_2$ and save results for later use.
3. Repeat another 8 steps and collect the results.

Now you have for each instance of your dataset, how it was classified, since the reunion of all $Test_i$ is the original data set (each group $G_i$ is tested once). You can measure how do you like the errors.

Now there are a couple of things which I believe you have to pay some attention. You said you have 20 target classes and 4000 samples. I do not know about your specific problem, but it does not seem to have plenty of data. So, I believe is better to do multiple cross validations and average the results, thus you decrease the chance to get too biased results.

Another thing to pay attention for is how do you build your folds. You might use simple random sampling, but I believe is better to use a stratified random procedure. Thus you increase the chances to have a usable CV estimation.

You might also consider bootstrap testing if you do not have enough instances for a 10-fold cross validation with stratified sampling.