2
$\begingroup$

I am doing a classification problem where I need to do 10-Fold Cross validation. At the end, I obtain accuracy results for 6 different models: KNN, SVM, Decision Tree, Random Forest, Multilayer Perceptron, and Gradient Boosting Ensemble.

My issue is that for each fold, for each model the accuracies are all very similar.

For example, for KNN I have the following accuracies, over 10 folds:

[0.8824006488240065, 0.8994322789943228, 0.8896999188969992, 0.8807785888077859, 
0.8848337388483374, 0.8953771289537713, 0.8872668288726683, 0.8969991889699919, 
0.8888888888888888, 0.8832116788321168]

Is the similarity in accuracies a normal result? My TA said they should be more different.

$\endgroup$
0

1 Answer 1

1
$\begingroup$

To add a bit to @usεr11852's answer:

  1. variance due to finite test sample size in each fold:

    A fraction of tested cases such as accuracy follows a binomial distribution. You can therefore do a rough check how much such variance you'd expect given the number of tested cases for each fold $n_{f}$: if the true accuracy is $p$, the variance of observed variance over test sets (folds) of size $n_{f} \approx \frac{1}{k} n_{total}$ is
    $$s^2(\hat p_f) = \frac{p (1 - p)}{n_f}$$

    • I use index $_f$ to indicate values referring to the folds, and
    • $\hat p$ to distinguish the observed proportion from its true value $p$.

    Plugging your fold accuracies into this as a back-of-the-envelope guesstimate, their variance would be consistent with sample sizes above roughly 2000 - 2500, i.e. 20k - 25k samples in total for a 10-fold CV scheme. If you had less cases, something is indeed wrong. (This guesstimate assumes variance from instability to be negligible compared to variance due to finite test sample size. Thus giving a lower bound on sample size.)

  2. classifier variability aka instability. There is nothing wrong with having very stable predictions, it is highly desirable.
    Indeed, cross validation assumes that the surrogate models of the folds are equivalent in their predictions (hence we're allowed to pool the 10 estimates) and moreover are equivalent to the model trained on the whole data set, whose performance we approximate by the (pooled) CV performance estimate.

    Still, your TA may know that for the data at hand and the training procedure you use, the models won't be very stable.

Both ways of your observed variance being too low point to different errors in your coding logic, so I'd recommend to get back to the TA about the details of the situation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.