4
$\begingroup$

Good morning, I have a question concerning a data set where I'm training some data mining models. i have 72 observation, so i can't split my data set into training set and test set and I want to use k fold cross validation. What is the best value for k in this case? I have tried K=3 but I'm not sure about the result, for example a classification tree is not deep in the training fold. thank you

$\endgroup$

3 Answers 3

5
$\begingroup$

As Dikran Marsupial says, LOO can be computed very cheaply for some models (and iff each case is exactly one row in the data matrix, i.e. these shortcuts don't work if you have repeated measurements of the same case that need to always be in the same subset).
OTOH, for some models LOO is known to have a large pessimistic bias (essentially because the tested case always belongs to the class that is underrepresented in the surrogate training set compared to the whole data set - stratified k-fold helps can solve this issue).

k-fold is usually a good choice (even better: repeated k-fold). Theoretically, the choice of k should not influence the results more than that small k may have more pessimistic bias (since the surrogate model training set gets considerably smaller than the whole data set). k between 5 and 10 are typically a good idea.

In your case, since 72 can be divided in 8 or 9 evenly-sized folds (no remainder), you may want to go with k = 8 or 9. But this is really not a critical choice.

$\endgroup$
3
$\begingroup$

Leave-one-out cross-validation is a common approach to this problem, mostly because it is cheap. If you pre-compute the matrix of distances between points and sort the columns (keeping a record of the indexes), then you can go down each column (starting from the second row as the first will be the distance to the left-out pattern itself), and work out what the classification will be for each value of k.

$\endgroup$
0
$\begingroup$

Good morning. As a general rule of thumb, k = 5 or k = 10 is a good compromise between bias and variance. However, it very much depends on the dataset. I would suggest you pick a model evaluation metric (e.g. accuracy, F-score, AUC ROC...) and plot it with respect to different values of k. Then you will know which k value is best.

$\endgroup$
1
  • $\begingroup$ Picking the "best" k is a very bad strategy: theoretically, we'd expect the observed performance to be almost independent of k (and with 72 cases certainly much lower than case-to-case variance and possibly the influence of model instability). Thus, we'd expect any such difference that points to a best k to be spurious. In consequence the better performance is overoptimistic bias. $\endgroup$ Jan 18 at 16:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.