I ran across a new paper from the Berkeley NLP group on statistical testing, An Empirical Investigation of Statistical Significance in NLP.

There is pseudocode for computing a p-value in the paper, basically, the idea is that the sample set of $x_1,x_2,...,x_N$ are sampled with replacement from data $x$. Then

$\text{p-value} = \text{count}(\delta(x_i) > 2\delta(x))/N$, where $\delta(x_i)$ is a metric gain.

I could understand the formula for computing the p-value in Koehn's paper Statistical significance tests for machine translation evaluation, in which:

$\text{p-value} = \text{count}(\delta_a(x_i) < \delta_b(x_i))/N$, where $\delta_a$ and $\delta_b$ are the metric gain for system $a$ and $b$ respectively.

Is there any explanation or reference for the formula $\text{p-value} = \text{count}(\delta(x_i) > 2\delta(x))/N$. The authors also noted that if the mean of $\delta(x_i)$ is $\delta(x)$ and $\delta(x_i)$ is symmetric, then both formulas above are equivalent.


1 Answer 1


As far as I understand from looking at section 2, the authors seem to explain their rationale for the bootstrap test as follows-

"the $x_i$ were sampled from $x$, and so their average $\delta(x_i)$ won’t be zero like the null hypothesis demands; the average will instead be around $\delta(x)$... The solution is a re-centering of the mean – we want to know how often $A$ does more than $\delta(x)$ better than expected. We expect it to beat $B$ by $\delta(x)$. Therefore, we count up how many of the $x_i$ have $A$ beating $B$ by at least $\delta(x)$."

The authors want to test if the gain is non-zero so they write the p-value as $\delta(x_i) < 2\delta(x)$ , which could be re-written as $0 < 2\delta(x) - \delta(x_i)$; because $E[\delta(x_i)]=\delta(x)$ the R.H.S. of the inequality then becomes $\delta(x)$, which is the $H_0$ they were seeking to reject.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.