This question is focused on seemingly difficult probability calculations in experiments for calculating p-values.

In the simple example of finding out if a 6 sided die is a fair or loaded die, one can easily calculate the probability of getting 10 1s in a row after rolling the die 10 times and determine if that value meets the requisite threshold for statistical significance.

However, how are problems approached that may have more difficult probability calculations? A 6 sided die is exceedingly simple, but other experiments may have more difficult probability calculations while assuming the null hypothesis.

For instance, does some new drug reduce the size of a cancer tumor. Would one have to gather a huge database of patients with cancer tumors and calculate over a 30 day period what the typical size change of such tumors is? But when designing the experiment to get the typical (non drug induced) change in tumors size, how does one know when that experiment gets to statistical significance? Since in that case we are AGAIN estimating a statistic from a sample of cancer patients.

It seems there is ample opportunity in a scientific process like the above to accumulate errors and thus distort the end result of whether such a drug works.

How is this issue handled in the real world.

  • 1
    $\begingroup$ This seems like a question relating to scientific approaches rather than statistical methods and philosophy. It is not a question about calculating a p-value. Can you make your question more answerable and more statistics-based? $\endgroup$ Nov 11 '21 at 6:52
  • $\begingroup$ Thanks for edit. Still a bit wordy, but two sentences make it clear the main issue is how to find P-values in various non-trivial situations. "This question is focused on seemingly difficult probability calculations in experiments for calculating p-values." ,,, "However, how are problems approached that may have more difficult probability calculations? " Hope my answer is helpful. $\endgroup$
    – BruceET
    Nov 11 '21 at 7:32
  • $\begingroup$ No links under 'Related' in the margin directly answer your question, but you may find some of them of interest, especially the much up-voted second link. $\endgroup$
    – BruceET
    Nov 11 '21 at 8:09
  • $\begingroup$ @H_1317 p-value calculations relate to some probability model under a null hypothesis -- precise details will depend on what your model is, and on your test statistic and the specific hypotheses. Given such details you compute the probability of getting a result at least as extreme (at least as far toward/into the rejection region) as the one for the sample, if the null hypothesis were true. $\endgroup$
    – Glen_b
    Nov 11 '21 at 10:26

Let's look at the P-value of a one-sided t test. Suppose we have data from $n=10$ observations from a population distributed $\mathsf{Norm}(\mu,\sigma),$ with $\mu$ and $\sigma$ both unknown. We want to test $H_0: \mu =20$ against $\mu < 20.$

Null hypothesis true. We might have data from a population for which $H_0$ is true, sampled using R, as follows:

x = rnorm(10, 20, 3)
[1] 18.85053

The sample mean $\bar X = 18.85$ is smaller than the hypothetical mean $\mu_0 = 20,$ but is it enough smaller that we reject $H_0$ at the 5% level of significance?

In R, this left-sided, one-sample t test looks like this:

t.test(x, mu = 20, alt="less")

        One Sample t-test

data:  x
t = -1.2168, df = 9, p-value = 0.1273
alternative hypothesis: 
 true mean is less than 20
95 percent confidence interval:
     -Inf 20.58225
sample estimates:
mean of x 

Assuming $H_0$ to be true, the test statistic $T$ is distributed according to Student's t distribution with $\nu = 10 - 1 = 9$ degrees of freedom. Its mean is $0.$

Our observed t statistic is negative: $T = -1.2168.$ The question is whether it is sufficiently negative to reject $H_0.$ Using R, we can find $P(T < -1.2168 | H_0 \mathrm{True}).$ [In R, pt is the CDF of a t distribution.]

pt(-1.2168, 9)
[1] 0.1273127

We see that the probability "under $H_0$" (assuming $H_0$ true) of a more extremely negative t statistic is fairly large $0.1273.$ More to the point, the probability exceeds $0.05 = 5\%,$ and so we do not reject $H_0.$ If you look at the output of the t test above, you will see that the P-value is given as $0.1273,$ which agrees with the computation using pt just above.

In the figure below, the curve is the density curve of the distribution $\mathsf{T}(\nu = 9).$ The vertical black bar is the observed value of the t statistic; the P-value is the area under the curve to the left of this line. The vertical dotted red bar cuts probability 5% from the lower tail of this distribution.

enter image description here

R code for figure:

hdr = "Density of T(9)"
curve(dt(x,9), -4, 4, lwd=2, ylab="Density", xlab="t", main=hdr)
abline(h=0, col="green2");  abline(v=0, col="green2")
abline(v=-1.1273, lwd=2)
abline(v=qt(.05,9), col="red", lty="dotted")

Null hypothesis false. By contrast, if I sample from a normal population with $\mu = 15,$ then there is a good chance I will reject the null hypothesis $H_0: \mu= 20$ in favor of the alternative $H_a: \mu < 20.$

y = rnorm(10, 15, 3)
[1] 16.02056

t.test(y, mu=20, alt="less")

        One Sample t-test

data:  y
t = -4.6489, df = 9, p-value = 0.000602
alternative hypothesis: 
 true mean is less than 20
95 percent confidence interval:
     -Inf 17.58971
sample estimates:
mean of x 

Now the P-value is very small. It would be almost impossible to get a test statistic as small or smaller than $T=-4.6489$ if $H_0$ were true.

pt(-4.6489, 9) 
[1] 0.0006019944

Again here, the P-value is the area under the density curve to the left of the vertical black line.

enter image description here

Note: R code similar to above for the figure is omitted.

Two-sided binomial test. Briefly, for an another example, suppose we wonder whether a coin is fair. We toss it $n = 100$ times to test $H_0: p = 0.5$ against the two-sided hypothesis $H_a: p \ne 0.5.$ Suppose I see $x = 62$ Heads in the 100 tosses.

Here is an exact binomial test. [The null hypothesis $H_0$ and a two-sided alternative are assumed, unless something else is specified.] The P-value is $0.02098 < 0.05 = 5\%,$ so the null hypothesis is rejected.

binom.test(x=62, n=100)

        Exact binomial test

data:  62 and 100
number of successes = 62, number of trials = 100, 
 p-value = 0.02098
alternative hypothesis: 
 true probability of success is not equal to 0.5
95 percent confidence interval:
 0.5174607 0.7152325
sample estimates:
probability of success 

For a two-sided test, in order to find the P-value, we need to look for the probability of a result as or more extreme than the one observed, but in both directions. So, assuming the coin is fair, we need to find $P(X \ge 62) + P(X \le 38) = 0.02098,$ as follows:

sum(dbinom(62:100, 100, .5)) + sum(dbinom(0:38, 100, .5))
[1] 0.02097874

[I have used R to get exact binomial probabilities, but for this problem a normal approximation would also give a useful answer.]

In the figure below, the P-value is the sum of the heights binomial probability bars outside the vertical blue lines.

enter image description here

R code for figure:

x = 0:100;  PDF = dbinom(x, 100, .5)
plot(x, PDF, type="h", lwd=2, main = "PDF of BINOM(100,.5)")
 abline(h=0, col="green2")
 abline(v=c(38.5, 61.5), col="blue")

Note: When you use a computer program to do a statistical test, the P-value is very often part of the output, so you usually don't need to compute P-values on your own. [It is not generally possible to use printed tables of t, chi-squared, F, and other distributions to get exact P-values. Typically, not enough probability values are included.]

In order to understand the use of P-values for statistical inference it is important to understand how P-values are computed.

  • $\begingroup$ My first concern is the assumption that is made at the beginning of all of these examples. Namely, we assume we know the probability distribution, whether that is normal or some other assumed distribution. Isnt that a bit of a leap? Second, and this is a computational question moreso, how can one calculate a probability of a normal distribution N(20,$\sigma$) where $\sigma$ is unknown? It would seem to me $\sigma$ is a necessary feature of the distribution for such a calculation (I would expect one could take an integral for <20 values of the pdf if $\sigma$ was known). $\endgroup$
    – H_1317
    Nov 12 '21 at 4:31
  • $\begingroup$ Often you have to rely on previous similar studies to know (or to guess) whether the data are normal or from some other family. And the answer to that will determine the type of test used, and hence the method of calculating P-values. $\endgroup$
    – BruceET
    Nov 12 '21 at 4:53

For complicated settings it is important to validate your modelling assumptions. If your distributional assumptions appear correct, then you have confidence that although you may have not selected the true distribution, you have adequately approximated the true distribution, and hence your p-values are still relevant.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.