I have a single observation with value $x = 0.5$ that comes out from a complicated computational process. I would like to know what is the probability to observe such value by chance.

To attempt to answer the question, I have ran the computational process with random input values for few thousand times. I get a distribution that is about normal, has mean very close to zero, and a standard deviation around 0.09.

Intuitively, it looks like the chance of observing 0.5 by chance is very little. However, how do I turn this into an actual statistical test?

  • $\begingroup$ I have posted code showing how to conduct permutation tests. A simple working example is at the end of the answer at stats.stackexchange.com/a/137467, for instance. A general description of permutation tests, along with a generic description of how to code them, appears in the survey at stats.stackexchange.com/a/104746. $\endgroup$
    – whuber
    Jul 5 '15 at 13:21

This is quite straightforward, there is no need to infer a distribution under the Null Hypothesis. Your p-value is just the number of times $x_{permuted}$ get superior or equal to $0.5$, divided by the number of permutations made.
This fits the definition of the p-value: "If H0 is true and a new sample is drawn, what is the probability to get at least such extreme results ?"

Maybe you mixed things up with the Bootstrap procedure which is generally done to estimate the "real" distribution of your statistic of interest.

I don't say your approach is completely wrong, if your distribution looks like normal, you could eventually do a z-test, and it should give a quite reliable p-value. But I think the spirit of the permutation test is just to count how often you indeed get equal or more extreme results because you have a direct access to it, whatever the real distribution of $x_{permuted}$ is.

For the sake of the comparison it would be interesting that you give how often $x_{permuted}$ get superior or equal to $0.5$ in your data set, we could compare it with what would give a 1-tailed z-test.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.