A widely cited paper "Statistical Significance Tests for Machine Translation Evaluation" proposes usage of Bootstrap for evaluating significance of difference across systems for Machine Translation (the domain is not important for the purpose of my question).

The authors describe their approach to Bootstrap hypothesis testing as follows:

Given a small collection of translated sentences, we repeatedly (say, 1000 times) create new virtual test sets by drawing sentences with replacement from the collection. For each set, we compute the evaluation metric score for both systems. We note, which system performs better. If, say, one system outperforms the other system 95% of the time, we draw the conclusion that it is better with 95% statistical significance. We call this method paired bootstrap resampling, since we compare a pair of systems.

According to my knowledge, statistical tests based on bootstrap (just like any statistical test) first specify null hypothesis (e.g. the two systems exhibit the same performance) and then draw samples according to this null hypothesis. Then we can measure, how unlikely was the difference between the two compared systems under this null hypothesis.

In the cited paper there is no null hypothesis explicitly stated and it is definitely not that systems are performing the same. It is rather finding confidence intervals for performance of each of the systems and then looking at their intersection.

My question is: is there any reason this method would be correct?

  • "According to my knowledge, statistical tests based on bootstrap ... first specify null hypothesis ... and then draw samples according to this null hypothesis. Then we can measure, how unlikely was the difference between the two compared systems under this null hypothesis." I think you're thinking of permutation tests. – Jake Westfall Oct 11 '15 at 16:16
  • This applies to any hypothesis test, bootstrap in particular. See Efron B., Tibshirani R. - An introduction to bootstrap (1993) page 221, where they talk about how the null hypothesis is being simulated from the data. – michal Oct 11 '15 at 16:23
  • I see what you are saying. A permutation test may possibly be more appropriate. However, if the bootstrap statistic (difference in performance) is asymptotically normal, the null of equal performance could be implicit, a 95% confidence interval could be formed and we could reject the null with the logic that realizing a difference as great or greater than the mean of the bootstrap given the null is less than 5%. But if the bootstrap statistic was not asymptotically normal (only a small set of discrete values possible for the difference) I am not sure if the same logic holds. – Zachary Blumenfeld Oct 12 '15 at 0:14
  • Sorry for a late reply, I was trying to understand. This is interesting, I am just a bit confused about the details. You are saying that we find the confidence interval for the difference of means using bootstrap and assume it is normal. Then, how exactly do we reason about the confidence interval to obtain p-value for our null hypothesis? – michal Oct 19 '15 at 16:19

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.