2
$\begingroup$

How do i compute the bootstrapped mean? I computed 3 validation accuracies using 3 fold accuracies, and wanted to compute the overall mean, or a bootstrapped mean?

Lets say i have these accuracies for the 3 folds being 0.1, 0.5 and 0.3 .

I understand the concept of bootstrapping, meaning to reuse data you already have, but how do i "reuse" the same data to compute the mean?..

Background:

So I am currently training a DNN acoustic model for speech recognition, and to ensure performance or accuracy of the validation accuracy, I need some idea of the of the range of which the accuracies lies in.

Usually would one do 10 fold CV, but given the amount of data and the time it would take for train all folds, I decided to go to the other way, and compute 3 fold CV, and bootstrapping the resulting distribution (an so called "accuracy distribution") => thereby extracting the mean, variance, and CI..

But doing that doesn't seem to be that simple or?

I initially thought that doing 10 fold CV and 3 fold CV with bootstrapping would in some sense give the same result, but that does not seem to be case?

$\endgroup$
4
$\begingroup$

Typically, you would not use bootstrapping to calculate the mean. Rather, the mean would be the empirical average from your original dataset, and the bootstrapping replicas (of which should there should be many more than 3, incidentally), would be used only to calculate the confidence interval of the mean.

One exception to this would be using bootstrapped bias correction (see Introduction To The Bootstrap, Or Michael Chernik's books), but, here too, you would start off with the mean from the original dataset. In each iteration, you would calculate the difference between the original mean, and the mean of the dataset. Using the frequencies for the biases, you could decide if the original mean is biased. Using bootstrapping for bias correction is dangerous, and is done more rarely.

Edit based on the question clarification

Here is the application of this to the specifics of your case. Suppose you divide your dataset into 3 parts, $A$, $B$, and $C$.

Using $A$ and $B$ you build a model, and predict $C'$; using $A$ and $C$ you build a model, and predict $B'$; using $B$ and $C$ you build a model, and predict $A'$.

Using $A, A'$, $B, B'$, and $C, C'$, you presumably build now $A''$, $B''$, and $C''$, indicating the cross-validated accuracy.

Define now $X = A'' + B'' + C''$, that is, $X$ is the concatenation of these three results. The estimated mean should be the mean of $X$. The confidence intervals should be calculated by bootstrapping $X$.

$\endgroup$
9
  • $\begingroup$ Just out of curiosity (and perhaps completeness sake), what danger of using bootstrapping for bias correction are you referring to? $\endgroup$ – IWS Jul 31 '17 at 12:39
  • 4
    $\begingroup$ @IWS Introduction To The Boostrap discusses this at the end of chapter 10. The variance of the bias-corrected estimate can be large. "Bias correction can be dangerous in practice. Even if $\bar{\theta}$ is less biased than $\hat{\theta}$, it may have substantially greater standard error." $\endgroup$ – Ami Tavory Jul 31 '17 at 12:43
  • $\begingroup$ When you say mean of the original dataset, are then implying the the average of the three folds or should train an entire new network, in which i use the complete dataset. I am not sure understand how you can calculate a CI for each accuracy fold, as these are very different.. $\endgroup$ – Bob Burt Aug 1 '17 at 12:39
  • $\begingroup$ @BobBurt I actually meant the mean of the entire dataset, without any folds. That's how bootstrapping methodology works. The folds are not for the statistic itself, just for it's CI. LMK if I misunderstood your question somehow. $\endgroup$ – Ami Tavory Aug 1 '17 at 18:59
  • 1
    $\begingroup$ @BobBurt Great, I'll have a look at it tomorrow. $\endgroup$ – Ami Tavory Aug 1 '17 at 21:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.