I was wondering how I can calculate p-value from a distribution of test statistic manually?


  • 1
    $\begingroup$ Are you familiar with the definition of a p-value? [If not, see the first sentence here.] That pretty much explains how to obtain it - you find the probability (under the null) of obtaining a test statistic at least as extreme as the sample test statistic, so if you have the null distribution, you find the relevant area. What counts as "at least as extreme" depends on your alternative. It's easier to explain if you ask about a specific example. $\endgroup$ – Glen_b Nov 4 '14 at 4:30
  • $\begingroup$ @ Glen_b how null distribution can be calculated for a given test statistic? $\endgroup$ – Alph Nov 4 '14 at 14:35
  • $\begingroup$ Your question above assumes you have the distribution. If you want to explore the very different issue of obtaining it, that would be a new question. $\endgroup$ – Glen_b Nov 5 '14 at 1:25
  • $\begingroup$ @ Glen_b so if the distribution is given then it always be null distribution ? $\endgroup$ – Alph Nov 5 '14 at 1:53
  • $\begingroup$ When you say in your question "from a distribution of test statistic", it's up to you to define what, exactly you mean by that. I took you to be talking about the null distribution because otherwise the question makes no sense (you can't work out p-values from things that aren't the null distribution!). You tell me what you're asking about in the original question, and then I can answer relative to what you're specifying. So is the question asking about the null distribution or not? $\endgroup$ – Glen_b Nov 5 '14 at 2:47

If your statistics is given by a probability density function $f(x)$ and you observe statistics $x_0$, then your p-value is typically $$\int_{x_0}^{\infty} f(x)~\mathrm{d}x$$ or $$\int_{-\infty}^{x_0} f(x)~\mathrm{d}x$$ depending on which side of the tail you're looking.

If instead your statistics is given as a sample, the p-value will be the quantile of $x_0$ in that sample.

  • $\begingroup$ @ Arthur B. Can you give me example of calculating p value when test statistic is sample? $\endgroup$ – Alph Nov 3 '14 at 22:50
  • 1
    $\begingroup$ It might be easy to simulate the statistics conditional on the null hypothesis. In this case, you could make 10,000 draws of the statistics and then compare where the statistics from your data set ranks. If it ranks in the top (or bottom, depending) 100 values, then you can reject the null hypothesis with p < 0.01 $\endgroup$ – Arthur B. Nov 3 '14 at 23:14
  • 1
    $\begingroup$ @Phil2014: In R, for example, if your test statistics are in the vector X, you would do something like quantile (X, c(0.025, 0.975)) (two-sided) or quantile (X, 0.95) (one-sided). $\endgroup$ – Wayne Nov 4 '14 at 16:49
  • $\begingroup$ @ Arthur B. Do we have to know the distribution? If we don't know the distribution, can we still get p-values? $\endgroup$ – Alph Apr 15 '15 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.