1
$\begingroup$

Question: Suppose that you are testing the idd-ness of a random number generator, and you've done so with the permutation test and the monkey test. Both tests produce a $\chi^2$ statistic and a corresponding $p$-value. How would you calculate the corresponding standard errors to indicate the accuracy of the statistic and the $p$-value?

I'm using python scipy.stats.chisquare. The function chisquare returns the $\chi^2$ statistic and the $p$-value, like so:

Power_divergenceResult(statistic=675.11538461538453, pvalue=0.49150891735031166)

However, these estimates don't tell me that much when I don't have a corresponding standard error, right?

$\endgroup$

1 Answer 1

2
$\begingroup$

Test statistics and p-values are random; they don't really estimate anything.

Under a point-null hypothesis the p-value (at least for a continuously distributed test statistic) should be uniformly distributed between 0 and 1.

[For discrete test statistics their cdf is a step function which touches the $y=x$ line at each jump, but the jumps might not be all the same height - they usually aren't.]

You seem to be expecting that the p-value and the chi-squared value should converge toward some particular value at large sample sizes (whence calculating a standard error might make sense). This is not the case.

To answer the direct question, the variance of the distribution of a p-value under the null (for a continuous test statistic) should be $1/12$. The standard deviation of the distribution of the p-value should then be the square root of that. Similarly, the variance of a chi-squared($k$) is $2k$, so the standard deviation is $\sqrt{2k}$.

[Under the alternative, p-values may tend to be typically small but are generally quite right skew. Under the alternative, the test statistic simply tends to be stochastically larger than under the null -- it is also not converging to some particular value. In the case of chi-square tests, typically under the alternative the test statistic has a noncentral chi-squared distribution, or approximately so. This may be the case for the tests you're looking at but I have not checked that it's so.]

I don't think there's much value in pursuing this line of thought, the distribution of the p-value isn't converging toward nor gathering about anything but 0, and then only under the alternative. The standard deviation of the distribution of the p-value under the null doesn't tell you anything; it's the same for every continuous test statistic (with a point null).

If you want to talk about accuracy in some useful sense then you should probably be looking at some kind of measure of effect size, or some confidence interval for whatever quantity your test is trying to pick up.

$\endgroup$
2
  • $\begingroup$ Thank you for your elaborate answer, this will help me a lot! Could you be so kind to explain to me why the variance of the distribution of the distribution isn't interesting to report? Suppose that the distribution under the null hypothesis has a very high variance, wouldn't it be kind of deceiving if we didn't report this? $\endgroup$ Commented Mar 19, 2018 at 10:02
  • $\begingroup$ Sure the distribution of the test statistic matters in how you understand a sample statistic. If I were to replace some positive test statistic $W$ with $T=e^W$, I would increase its variance ... but change nothing relevant about the test other than the distribution of the statistic. Knowing where some observed test statistic $w$ occurs in the distribution of $W$ is the same as knowing where $t=e^w$ is in the distribution of $T$ (they're at the same percentile) -- they convey the same information about how consistent the sample is with the null hypothesis. $\endgroup$
    – Glen_b
    Commented Mar 19, 2018 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.