Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite

I have 2 ASR (Automatic Speech Recognition) models, providing me with text transcriptions for my testdata. The error measure I use is Word Error Rate.

What methods do I have to test for statistical significance of my new results?

An example:

I have an experiment with 10 speaker, all having 100 (the same) sentences, total 900 words per speaker. Method A has an WER (word error rate) of 19.0%, Method B 18.5%.

How do I test whether Method B is significantly better?

share|improve this question

3 Answers

up vote 2 down vote accepted

The Sclite tool from NIST offers a statistical test to compare two ASR systems on the same test set (http://www.itl.nist.gov/iad/mig//tools/).

For the test you described several of the test offered would be suitable (including the sign test) but not all are equally powerful.

share|improve this answer
When I initially asked this question I was planning to add this answer myself, besides alternatives I didn't know. Thanks for answering, I will accept this answer as the accepted as it is a 'standard' in the ASR world! – Peter Smit Nov 29 '10 at 19:12
up vote 2 down vote

Suppose that the text has N words and that you require that an ASR should correctly predict at least 95% of words in the text. You currently have the observed error rate for the two methods. You can perform two type of tests.

Test 1: Do the ASR models meet your criteria of 95% prediction?

Test 2: Are the two ASR models equally good in speech recognition?

You could make different type of assumptions regarding the data generating mechanism for your ASR models. The simplest, although a bit naive, would assume that word detection of each word in the text is an iid bernoulli variable.

Under the above assumption you could do a test of proportions where you check if the error rate for each model is consistent with a true error rate of 5% (test 1) or a test of difference in proportions where you check if the error rates between the two models is the same (test 2).

share|improve this answer
I don't have any requirements how correct my ASR should be. For example, when I change my model a bit my error goes down from 30% to 29.8%. For that change I want to now if it's significant – Peter Smit Jul 21 '10 at 6:17
In that case you would use test 2. See the table for "Two-proportion z-test, pooled for d0 = 0" at the wiki: en.wikipedia.org/wiki/… – user28 Jul 21 '10 at 10:44
up vote 0 down vote

Assuming there is some training involved, you may use some kind of cross-validation, or bootstrap of a train set.
If not, stick to the Srikant solution. I would do it even simpler, just assuming that the number of error is Poisson distributed.

share|improve this answer
There is a lot of training involved. Training a reasonable Finnish acoustic model can take 60-100 hours of computing time. Besides that, most corpora work with standardized training, development and evaluation sets. – Peter Smit Jul 24 '10 at 4:50
E, 100 cores and one may live with that. Seriously, this looks dubious from a ML point of view, still I can understand the reasons why you do it in such a way. So then stick to the Srikant solution. – mbq♦ Jul 24 '10 at 8:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.