Non-inferiority test and bootstrapping

Question

Background: I am trying to show that my method is as good as or better than a previously published work in the field of Automatic Speech Recognition.

Usually, in the literature people usually report the word error rate (what fraction of words the recognition system got wrong) on a few sets of utterances and that's it. However, I would like to provide a more statistically-sound analysis of my results performing a non-inferiority test. It is important to remark that I am not claiming that my method is more accurate than the previous method, but it's (statistically) significantly faster and non-inferior. Thus, the non-inferiority test instead of the traditional hypothesis testing.

Problem: Word error rates are typically measured globally over a set of utterances: we compute the number of errors across all samples of our dataset and then divide by the total number of words in the reference transcription. Thus, for a particular dataset, I do not have a mean and a standard deviation.

Attempt at Solution: However, I thought I could use bootstrapping to estimate the mean and variance of the word error rate of both methods: I compute the errors on each of my $n$ utterance samples, and then I create $r$ bootstrap samples of size $n$ and compute the global error rate for each of them, in order to compute the mean and standard errors.

Now, I can try to perform the following non-inferiority test: $$ H_0 : \text{Error}_{New} - \text{Error}_{Old} \geq \delta \\ H_a : \text{Error}_{New} - \text{Error}_{Old} < \delta \\ $$

However, I have problems computing the t-statistic, since I am not sure of the degrees of freedom of my method. My original dataset was made of $n$ samples from which I computed the word error rate, however my estimate of the mean and the variance of the word error rate was computed from $r$ bootstrap samples.

Question: So, what should be the number of degrees of freedom? $r - 1$, $n - 1$, or neither and my approach is totally wrong?

Some actual numbers:

• New method's global error rate: 5.8
• Old method's global error rate: 5.9
• Bootstrapped mean/std. error of the new method: 5.751 / 0.193
• Bootstrapped mean/std. error of the old method: 5.876 / 0.177
• Bootstrapped mean/std. error of $\text{Error}_{New} - \text{Error}_{Old}$: -0.125 / 0.145
• $n$ = 966
• $r$ = 3000

I should mention that, regardless of using $r-1$ or $n-1$ as my number of d.o.f., I got a p-value $\approx 0$ (even if I use $\delta = 0$), but I would like to know which number should I actually use as the number of d.o.f. and whether or not my approach is statistically sound, or perhaps I am making some mistake.

Answer 1

If the errors are truly independent, could you treat this as a proportion test instead?

That is, if, across all samples, your program made 58 errors and the old program made 59 errors across 1,000 words (or whatever number you use to get to 5.8 and 5.9 error rate) and run a proportion test.

If the analyses are from separate recordings with different difficulties/underlying error-rates, you could use a binomial model with both method and sample as factors.

Assuming that you used the exact same samples for both methods (I would assume this approach), you could apply truly paired analyses. This would be particularly relevant if the errors tend to be on the same utterances (e.g., both programs mistake the same word). Specifically, I think that McNemar's Test would be the most appropriate here.