# Computing p-values

I am a computer engineer, living in a digital world. I have no probabilistic background besides a probability course in the University and tried to grasp the Fisher's idea and have got the following questions.

Why is this room called Ten Fold and Corss validation? Wikipedia says that "To compute a p-value from the test statistic, one must simply sum (or integrate over) the probabilities of more extreme events occurring". Thereby, we are warned against interpreting p-values as type I errors. But, AFAIK, $a$-values-based Type I errors just mean exactly that you cut the tail distribution whereas correct result resides in it, which is exactly Type 1 error.

Secondly, I could not find the formulas or examples computing p-values. Instead WP tells that they were invented by Fisher, who Rather than using a table of p-values, inverted the CDF, publishing a list of values of the test statistic for given fixed p-values. Does it say that I must know the p-value in advance? How do I compute it if I must know it in advance?

Third, Wikipedia gives a hint on a way to compute the p-value: "Hypothesis tests, such as Student's t-test, typically produce test statistics whose sampling distributions under the null hypothesis are known. To compute a p-value from the test statistic, one must simply sum (or integrate over) the probabilities of more extreme events occurring." This begs a question: do we need convert t- into p-value and apply Fisher test if we can use the t-value and Student test immediately? Why is Fisher's approach better?

Finally, Wikipedia says that the formula for p-value is $2/2^n$. But, it is stated in coin flipping context and it is not clear if it is true in every context. Thereby, it points out that rolling two dies, which yield 6+6=12 has probability of 1/36, which is sufficient to overcome the 5% threshold but it is not statistically significant. This leaves me with uncertainty: how can p-value determine the statistical significance after that?

A $p$-value is the probability of someone getting a test statistic as far from 0 as it is observed or farther, given the sample size at hand. It is not related with type I errors. To use MacKay's definition in his book Information Theory, Inference, and Learning Algorithms:

p-value is the probability, given a null hypothesis for the probability distribution of the data, that the outcome would be as extreme as, or more extreme than, the observed outcome.

Here is a tiny example of computing $p$-values in the case of a simplistic linear model:

set.seed(10);
x = 1:20;
y = 2*x + rnorm(20);
lm1 <- lm(y ~ x)
summary(lm1)
Call:
lm(formula = y ~ x)

Residuals:
Min       1Q   Median       3Q      Max
-1.49158 -0.35334 -0.02612  0.59079  1.13746

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -0.58248    0.35503  -1.641    0.118
x            2.04971    0.02964  69.160   <2e-16 ***
Residual standard error: 0.7643 on 18 degrees of freedom


To restate our problem with actually numbers: the $p$ value (<2e-16) is the probability that the value of our test statistic (t value here) will be "as extreme as, or more extreme than" the observed value (69.160), given our null hypothesis ($t$-distribution).

So we want the $Prob(x>|69.160|)$. Fine; that probability requires a distribution to be meaningful. Fine again; will use a $t$-distribution because we are using a $t$-test after all. $t$-distributions require degrees of freedom(DF) though. Fine once more, we have the DF's of our model (here the number of data points minus the number of model parameters) 18. So, we want pt(df=18,-69.160)? Almost, remember we want the absolute in this case (we are looking at a two-way test) so 2*pt(df=18,-69.160)? Yeap. Pretty much. (I hope you understand R-syntax :D )

2*pt(df=18,-69.160) < 2e-16
[1] TRUE


So no, you don't know the p-value in advance you calculate it given you model at hand. There are certain tables of $p$-values that are handy when you are not having access to a computer but in general nowadays you can get $p$-values on the fly. This link http://www.cyclismo.org/tutorial/R/pValues.html is rather helpful actually.

For your final question: The null hypothesis of a fair dice means you have a uniform (discrete) distribution $[1,6]$ for a single dice. And a non-uniform distribution between $[2,12]$ (you can convince yourself for that; you have 36 different roll outcomes and only a single one yields a value of 12). As such a single sample from that distribution to equate to 12 appears is unlikely ($1/36$). And based on that you would reject the claim that your dices are fair (aka. you would reject your null-hypothesis). What wikipedia then says is that you totally ignore sample size effects if you do that. It would be wrong to use a $p$-value generated this way to determine the statistical significance of your null hypothesis. You need more rolls (samples) to assume as your expected roll outcome and test it against your null hypothesis.

Just to restate the take-home message. Given you decide your model/null hypothesis the $p$-value will be the probability of your test statistic to be as extreme as, or more extreme than, what you observed for your sample at hand. If your sample is not large enough to approximate the population behaviour (as for instance your single two-dice roll case) statistical inference will be meaningless (if not outright wrong).

• Excellent answer. I was confused about how to calculate the t value in your example. However, Wikipedia explains why: en.wikipedia.org/wiki/…. In R code: denom <- sqrt((1/(length(x)-2))*sum(resid(lm1)^2)) / sqrt(sum((x-mean(x))^2)); coef(lm1)['x']/ denom. This is equal to 69.1599 Commented Sep 28, 2013 at 19:06

I will try for a more intuitive explanation:

Consider the case where you are asked to compare 2 distances to see which is further, but one of the distances is given to you in miles and the other in kilometers. You cannot compare the 2 numbers directly, but first need to convert at least one of them. But, it does not matter if you convert the kilometers to miles and compare miles to miles, or convert miles to kilometers and compare kilometers to kilometers, or even convert both to leagues and compare leagues to leagues. As long as all the conversions are correct we will come to the same conclusion as to which distance is further.

The same is true with hypothesis testing, on one end we have the data, on the other end we have our type I error rate ($\alpha$), and between we have hypotheses and assumptions.

Simplifying a bit there are 2 directions of conversion:

data -> test statistic -> P-value

???? <- critical value <- alpha ($\alpha$)

We can compare a p-value to alpha to declare significance, or we can compare the test statistic to the critical value. Sometimes we can even compare a statistic that is not the usual test statistic to a different critical value (the ???? above). The hypothesis and assumptions determine which test statistic we compute and how we convert either from test statistic to p-value or from alpha to critical value.

For a more concrete example lets say that we want to test if a coin is fair by flipping it 10 times. The null hypothesis is that the probability of heads is 0.5 and the alternative is that it is not 0.5 in either direction. We can do the test using the binomial distribution, or the normal approximation to the binomial (probably not a good idea with n=10, possibly reasonable if we flipped the coin 20 times, would be fine if we flip it 100 times) or a couple of other methods (this is why you sometimes see computations using the t distribution or the f distribution or the ? distribution).

In this example there are only 11 possible outcomes (number of heads out of 10 flips), so if we use an alpha of 0.05 then the true probability of a type I error is about 0.021 (probability of flipping 0, 1, 9, or 10 heads). If we now do the experiment and get 7 heads then we would calculate the p-value as the probability of getting 7, 8, 9, or 10 heads plus the probability of getting 3, 2, 1, or 0 heads since those are as extreme in the opposite direction. This p-value is 0.344 and we would compare that to our alpha of 0.05 and not reject the null hypothesis.

Or we could have looked at the probabilities of the different outcomes when the null is true and created a critical region. We do this by starting at the extremes and working in as far as we can without exceeding alpha. In this case the critical region says we will reject if we see 0, 1, 9, or 10 heads and not reject if we see other values. Then we flip the coin and see 7 heads (same as above) and from that we don't reject, same conclusion.

Either way will give us the same results, we will see a p-value < 0.05 if we flip 0, 1, 9, or 10 heads and a p-value > 0.05 otherwise. Personally the critical value approach is more intuitive to me, but computers have made the p-value approach more common (and there are advantages there as well).

Why do we start from the extremes when calculating the critical region, or why do we include all the more extreme values in calculating the p-value? Well consider this case, with 10 flips of a fair coin there is a 0.0439 probability of flipping exactly 8 heads, so we could define our critical region as flipping 8 heads. But if we expand the region to 8 or 9 heads then the probability is greater than 0.05 (0.0537 to be more precise), so that does not work. We could set the rejection region to be 0, 8, or 10 heads and that keeps us below 0.05 (0.0459). But does that really make sense as a critical region? before flipping the coin we are already saying that if we see 0, 8, or 10 heads we will reject the null, but if we see 9 heads (or 1 head) we will not reject the null, this works mathematically but not intuitively that if 8 is extreme enough then 9 should be even more evidence, not less.