A widely cited paper "Statistical Significance Tests for Machine Translation Evaluation" proposes usage of Bootstrap for evaluating significance of difference across systems for Machine Translation (the domain is not important for the purpose of my question).
The authors describe their approach to Bootstrap hypothesis testing as follows:
Given a small collection of translated sentences, we repeatedly (say, 1000 times) create new virtual test sets by drawing sentences with replacement from the collection. For each set, we compute the evaluation metric score for both systems. We note, which system performs better. If, say, one system outperforms the other system 95% of the time, we draw the conclusion that it is better with 95% statistical significance. We call this method paired bootstrap resampling, since we compare a pair of systems.
According to my knowledge, statistical tests based on bootstrap (just like any statistical test) first specify null hypothesis (e.g. the two systems exhibit the same performance) and then draw samples according to this null hypothesis. Then we can measure, how unlikely was the difference between the two compared systems under this null hypothesis.
In the cited paper there is no null hypothesis explicitly stated and it is definitely not that systems are performing the same. It is rather finding confidence intervals for performance of each of the systems and then looking at their intersection.
My question is: is there any reason this method would be correct?