What method to use to test Statistical Significance of ASR results I have 2 ASR (Automatic Speech Recognition) models, providing me with text transcriptions for my testdata. The error measure I use is Word Error Rate.
What methods do I have to test for statistical significance of my new results?
An example:
I have an experiment with 10 speaker, all having 100 (the same) sentences, total 900 words per speaker. Method A has an WER (word error rate) of 19.0%, Method B 18.5%.
How do I test whether Method B is significantly better?
 A: The Sclite tool from NIST offers a statistical test to compare two ASR systems on the same test set (http://www.itl.nist.gov/iad/mig//tools/).
For the test you described several of the test offered would be suitable (including the sign test) but not all are equally powerful.
A: Suppose that the text has N words and that you require that an ASR should correctly predict at least 95% of words in the text. You currently have the observed error rate for the two methods. You can perform two type of tests.
Test 1: Do the ASR models meet your criteria of 95% prediction?
Test 2: Are the two ASR models equally good in speech recognition?
You could make different type of assumptions regarding the data generating mechanism for your ASR models. The simplest, although a bit naive, would assume that word detection of each word in the text is an iid bernoulli variable.
Under the above assumption you could do a test of proportions where you check if the error rate for each model is consistent with a true error rate of 5% (test 1) or a test of difference in proportions where you check if the error rates between the two models is the same (test 2). 
A: Assuming there is some training involved, you may use some kind of cross-validation, or bootstrap of a train set.
If not, stick to the Srikant solution. I would do it even simpler, just assuming that the number of error is Poisson distributed.
