4
$\begingroup$

This is a very basic question about cross-validation. Say that I have a sample size of 2901(or any difficult to divide number). How do I split this up into equal partitions (other than n=1)? And how big should I make each partition?

For example, if I make each partition size 300 (which gives me approximately 10 partitions), I will have some data points that are in more than one partition, giving it an unfair weight. Is this acceptable/what do people normally do about this?

By the way, I wanted to split it into equal partitions so that I can easily write code that will perform cross validation for any number of partitions.

$\endgroup$
2
$\begingroup$

You could also draw a sample of size 2901 from a uniform value taking its values in {1, 2, ... 10} (for 10-CV). This sample determines a random partition of the data, with 10 sub-groups of roughly equal size (each sub-group has equal probability of being drawn for each training observation).

For each of these sub-groups you can then compute a sub-group average measure (such as MSE, as opposed to total sub-group squared error) as suggested by Matthew.

I wouldn't think small differences in subsamble sizes would matter much if you take sub-group averages, but if you are worried you can always repeat this sampling procedure multiple times to get an average 10-CV estimate.

$\endgroup$
  • $\begingroup$ I like this solution. It seems that it could add significantly to my storage needs if the sample size is much larger than 2901. Basically, you are adding an extra column of data (if each sample is a row). This is probably not big enough of a problem to really worry about it. $\endgroup$ – jyfan Jul 26 '14 at 1:31
  • $\begingroup$ Exactly, 2901 new values is nothing for a computer. $\endgroup$ – jubo Jul 26 '14 at 16:02
1
$\begingroup$

A lot of validation measures are usually 'averaged' over the whole partition to allow for direct comparison on other data sets. For example, mean square error for continuous prediction is

$$ \text{MSE} = \frac{1}{N} \sum_{i=1}^N (\text{prediction}_i - \text{actual}_i)^2. $$

Other examples are MAE, area under the ROC curve, Brier score, R squared (generalized or otherwise). In this way, if the difference in size between partitions is small (say, within a few) then you shouldn't be worried about any kind of imbalance. In your case for 10-fold CV, (after shuffling the data to ensure random assignment) I would take $\lfloor 2901/10 \rfloor = 290$-sized partitions and give the last data point to any of the partitions. It won't matter which.

You can always make your validation measure size-independent by dividing by the partition size. For example, if you calculate logarithmic scoring in total, then just divide the total score by the partition size.

$\endgroup$
  • $\begingroup$ I think I will probably end up shuffling all my data first and then do the 2901/n with remainder randomly assigned to partitions. $\endgroup$ – jyfan Jul 26 '14 at 1:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.