# What method to use to test Statistical Significance of ASR results

I have 2 ASR (Automatic Speech Recognition) models, providing me with text transcriptions for my testdata. The error measure I use is Word Error Rate.

What methods do I have to test for statistical significance of my new results?

An example:

I have an experiment with 10 speaker, all having 100 (the same) sentences, total 900 words per speaker. Method A has an WER (word error rate) of 19.0%, Method B 18.5%.

How do I test whether Method B is significantly better?

The Sclite tool from NIST offers a statistical test to compare two ASR systems on the same test set (http://www.itl.nist.gov/iad/mig//tools/).

For the test you described several of the test offered would be suitable (including the sign test) but not all are equally powerful.

• When I initially asked this question I was planning to add this answer myself, besides alternatives I didn't know. Thanks for answering, I will accept this answer as the accepted as it is a 'standard' in the ASR world! Nov 29 '10 at 19:12

Suppose that the text has N words and that you require that an ASR should correctly predict at least 95% of words in the text. You currently have the observed error rate for the two methods. You can perform two type of tests.

Test 1: Do the ASR models meet your criteria of 95% prediction?

Test 2: Are the two ASR models equally good in speech recognition?

You could make different type of assumptions regarding the data generating mechanism for your ASR models. The simplest, although a bit naive, would assume that word detection of each word in the text is an iid bernoulli variable.

Under the above assumption you could do a test of proportions where you check if the error rate for each model is consistent with a true error rate of 5% (test 1) or a test of difference in proportions where you check if the error rates between the two models is the same (test 2).

• I don't have any requirements how correct my ASR should be. For example, when I change my model a bit my error goes down from 30% to 29.8%. For that change I want to now if it's significant Jul 21 '10 at 6:17
• In that case you would use test 2. See the table for "Two-proportion z-test, pooled for d0 = 0" at the wiki: en.wikipedia.org/wiki/…
– user28
Jul 21 '10 at 10:44

Assuming there is some training involved, you may use some kind of cross-validation, or bootstrap of a train set.
If not, stick to the Srikant solution. I would do it even simpler, just assuming that the number of error is Poisson distributed.

• There is a lot of training involved. Training a reasonable Finnish acoustic model can take 60-100 hours of computing time. Besides that, most corpora work with standardized training, development and evaluation sets. Jul 24 '10 at 4:50
• E, 100 cores and one may live with that. Seriously, this looks dubious from a ML point of view, still I can understand the reasons why you do it in such a way. So then stick to the Srikant solution.
– user88
Jul 24 '10 at 8:44