0
$\begingroup$

Does minimising the standard deviation of CV folds have any correlation with model accuracy as a theme?

I've noticed that changing the order of rows in a training data can change the AUC for each CV fold and hence the standard deviation of the mean CV. It seems that minimising the deviation results in having each fold being more representative of the population. So the question above might be stated: does training a model on more representative samples increase accuracy. If so, then the sort order of the data set might be interpreted as another tuning parameter.

$\endgroup$

1 Answer 1

2
$\begingroup$

I would not recommend optimizing your split order.

  • Observed variance in cross validation estimates is not a suitable surrogate for measuring representativity.

  • At least for figures of merit like accuracy, sensitivity, specificity and the like, this will push the splitting in direction of finding "easy" splits - which is quite different from finding representative splits.

    Accuracy & Co. have low variance when close to 0 or 1. So pushing for low variance is the same as looking for splits that have high accuracy.

  • What you can do, though, is use stratified splitting based on external knowledge about confounding factors/groups in the data in order to ensure representativity.
    This is the only honest way to optimize representativity of your splitting I can think of right now.


Variance in resampling verification results is comes from several sources:

  • general case-to-case variation (basically residual error): you only way to reduce this is having more test cases overall.
  • instability of the surrogate models adds further variance. This would be something you want to minimize. But via model training, not by finding easy-to-predict splits. I.e., model complexity vs. no. of training cases.

  • the data at hand being a sample from the general population. Within that one data set, there's nothing you can do about this. But then, it matters only if you want to compare algorithms, if your main objective is the model you can train from the data at hand, you're mostly fine.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.