2
$\begingroup$

I’m confused by using one standard error rule in repeated k fold scenario. In k fold cross validation, the standard error is of the error metrics is calculated as:

SEerr=SD(ERR)/sqrt(k)

which ERR is the error metric, k is the data split number. But in repeated k fold, if we just calculate standard error as SEerr=SD(ERR)/sqrt(k*repeat), then the SE is dependent on repeat number. The higher the repeat, the lower the standard error we can get. This calculation violates my intuition, and I think it must violates some statistical principles like independence et.al.

$\endgroup$
2
  • 1
    $\begingroup$ Standard error in general is observed standard deviation (not mean) / number of observations. $\endgroup$ – cbeleites unhappy with SX Feb 10 '20 at 11:59
  • $\begingroup$ Sorry, I've modified the question. $\endgroup$ – Jiahao-AI Feb 10 '20 at 15:50
2
$\begingroup$

IMHO your confusion is spot on.

The difficulty arises from the fact that we mix variance from two different sources here into one standard deviation (or variance).

  1. The number of independent tested cases determines how certain we can be about the performance of one particular surrogate model.
    I'll refer to this variance as $\sigma^2_n$.
  2. But there may be variance between the surrogate models = model instability as well.
    I'll use $\sigma^2_i$ for this.

Now we may really want to pull these two variances apart because they have different implications for our hyperparameter optimization. High $\sigma^2_i$ means that we need to change our models towards more stable training. High $\sigma^2_n$ means that we'd need to get more cases in order to say more.

The observed variance across the $r$ full repetitions (so $k$ folds pooled) is $\sigma^2_r = \frac{1}{k} \sigma^2_i$ and the observed variance across all folds (of all repetitions) is $\sigma^2_f = \frac{k}{n} \sigma^2_n + \sigma^2_i$. This allows to calculate $\sigma^2_n$ and $\sigma^2_i$ (there may be better ways to estimate $\sigma^2_i$ and $\sigma^2_n$, though).

I'm currently working on this I'm currently working on this (for different loss functions, and on optimization heuristics that use both variances), but it is not yet ready.


Meanwhile, I use the following heuristic:

The actual sample size of course doesn't improve by repeating the cross validation. So iff the models are sufficiently stable so that $\sigma^2_i \ll \sigma^2_n$, i.e. sample size is the limiting factor, the denominator of the standard error calculation will stay $\sqrt{k}$: repeating cross validation doesn't add any new (independent) test cases, we've seen them all in the first $k$ folds.

For the data sets I work with, the limiting factor is indeed almost always the number of independent cases. So unless the hyperparameter optimization contains heavily overfitting regions, I'm fine with this.

It is also easy to check whether stability is OK in the apparent optimum (by looking at the sd of the predictions for the same case across the repetitions). If that is not the case (and also not for lower complexity), I anyways have to go back and think about aggregating, so a drastically changed modeling approach.

An analogue heuristic may be built if model instability is the limiting factor.

$\endgroup$
2
  • $\begingroup$ Thank you. Can you give some guidance on how to implement the one standard error rule in repeated k fold scenario? Or some references? $\endgroup$ – Jiahao-AI Feb 10 '20 at 15:55
  • $\begingroup$ @Jiahao-AI: the "Meanwhile, I use the following" heuristic uses repeated k-fold and the one standard error rule. References wrt. what? Model stability? One-se-rule? Different sources of variance for test results? $\endgroup$ – cbeleites unhappy with SX Feb 14 '20 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.