I have 2 ASR (Automatic Speech Recognition) models, providing me with text transcriptions for my testdata. The error measure I use is Word Error Rate.

What methods do I have to test for statistical significance of my new results?

An example:

I have an experiment with 10 speaker, all having 100 (the same) sentences, total 900 words per speaker. Method A has an WER (word error rate) of 19.0%, Method B 18.5%.

How do I test whether Method B is significantly better?


The Sclite tool from NIST offers a statistical test to compare two ASR systems on the same test set (http://www.itl.nist.gov/iad/mig//tools/).

For the test you described several of the test offered would be suitable (including the sign test) but not all are equally powerful.

  • $\begingroup$ When I initially asked this question I was planning to add this answer myself, besides alternatives I didn't know. Thanks for answering, I will accept this answer as the accepted as it is a 'standard' in the ASR world! $\endgroup$
    – Peter Smit
    Nov 29 '10 at 19:12

Suppose that the text has N words and that you require that an ASR should correctly predict at least 95% of words in the text. You currently have the observed error rate for the two methods. You can perform two type of tests.

Test 1: Do the ASR models meet your criteria of 95% prediction?

Test 2: Are the two ASR models equally good in speech recognition?

You could make different type of assumptions regarding the data generating mechanism for your ASR models. The simplest, although a bit naive, would assume that word detection of each word in the text is an iid bernoulli variable.

Under the above assumption you could do a test of proportions where you check if the error rate for each model is consistent with a true error rate of 5% (test 1) or a test of difference in proportions where you check if the error rates between the two models is the same (test 2).

  • $\begingroup$ I don't have any requirements how correct my ASR should be. For example, when I change my model a bit my error goes down from 30% to 29.8%. For that change I want to now if it's significant $\endgroup$
    – Peter Smit
    Jul 21 '10 at 6:17
  • $\begingroup$ In that case you would use test 2. See the table for "Two-proportion z-test, pooled for d0 = 0" at the wiki: en.wikipedia.org/wiki/… $\endgroup$
    – user28
    Jul 21 '10 at 10:44

Assuming there is some training involved, you may use some kind of cross-validation, or bootstrap of a train set.
If not, stick to the Srikant solution. I would do it even simpler, just assuming that the number of error is Poisson distributed.

  • $\begingroup$ There is a lot of training involved. Training a reasonable Finnish acoustic model can take 60-100 hours of computing time. Besides that, most corpora work with standardized training, development and evaluation sets. $\endgroup$
    – Peter Smit
    Jul 24 '10 at 4:50
  • $\begingroup$ E, 100 cores and one may live with that. Seriously, this looks dubious from a ML point of view, still I can understand the reasons why you do it in such a way. So then stick to the Srikant solution. $\endgroup$
    – user88
    Jul 24 '10 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.