What is the meaning of p values and t values in statistical tests?

Question

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It seems that students easily learn how to perform the calculations required by a given test but get hung up on interpreting the results. Many computerized tools report test results in terms of "p values" or "t values".

How would you explain the following points to college students taking their first course in statistics:

• What does a "p-value" mean in relation to the hypothesis being tested? Are there cases when one should be looking for a high p-value or a low p-value?

• What is the relationship between a p-value and a t-value?

Answer 1

Understanding p-value

Suppose, that you want to test the hypothesis that the average height of male students at your University is 5 ft 7 inches. You collect heights of 100 students selected at random and compute the sample mean (say it turns out to be 5 ft 9 inches). Using an appropriate formula/statistical routine you compute the p-value for your hypothesis and say it turns out to be 0.06.

In order to interpret p=0.06 appropriately, we should keep several things in mind:

1. The first step under classical hypothesis testing is the assumption that the hypothesis under consideration is true. (In our context, we assume that the true average height is 5 ft 7 inches.

2. Imagine doing the following calculation: Compute the probability that the sample mean is greater than 5 ft 9 inches assuming that our hypothesis is in fact correct (see point 1).

In other words, we want to know P(Sample mean >= 5 ft 9 inches | Given true value = 5 ft 7 inches).

The calculation in step 2 is what is called the p-value. Therefore, a p-value of 0.06 would mean that if we were to repeat our experiment many, many times (each time we select 100 students at random and compute the sample mean) then 6 times out of 100 we can expect to see a sample mean greater than 5 ft 9 inches.

Given the above understanding, should we still retain our assumption that our hypothesis is true (see step 1). Well, a p=0.06 indicates that one of two things have happened:

(A) Either our hypothesis is correct and an extremely unlikely event has occurred (e.g., all 100 students are student athletes)

or

(B) Our assumption is incorrect and the sample we have obtained is not that unusual.

The traditional way to choose between (A) and (B) is to choose an arbitrary cut-off for p. We choose (A) if p > 0.05 and (B) if p < 0.05.

PS: Need to run now. Will add about the t-test when I find some more time.

Answer 2

No amount of verbal explanation or calculations really helped me to understand at a gut level what p-values were, but it really snapped into focus for me once I took a course that involved simulation. That gave me the ability to actually see data generated by the null hypothesis and to plot the means/etc. of simulated samples, then look at where my sample's statistic fell on that distribution.

I think the key advantage to this is that it lets students forget about the math and the test statistic distributions for a minute and focus on the concepts at hand. Granted, it required that I learn how to simulate that stuff, which will cause problems for an entirely different set of students. But it worked for me, and I've used simulation countless times to help explain statistics to others with great success (e.g., "This is what your data looks like; this is what a Poisson distribution looks like overlaid. Are you SURE you want to do a Poisson regression?").

This doesn't exactly answer the questions you posed, but for me, at least, it made them trivial.

Answer 3

Before touching this topic, I always make sure that students are happy moving between percentages, decimals, odds and fractions. If they are not completely happy with this then they can get confused very quickly.

I like to explain hypothesis testing for the first time (and therefore p-values and test statistics) through Fisher's classic tea experiment. I have several reasons for this:

(i) I think working through an experiment and defining the terms as we go along makes more sense that just defining all of these terms to begin with. (ii) You don't need to rely explicitly on probability distributions, areas under the curve, etc to get over the key points of hypothesis testing. (iii) It explains this ridiculous notion of "as or more extreme than those observed" in a fairly sensible manner (iv) I find students like to understand the history, origins and back story of what they are studying as it makes it more real than some abstract theories. (v) It doesn't matter what discipline or subject the students come from, they can relate to the example of tea (N.B. Some international students have difficulty with this peculiarly British institution of tea with milk.)

[Note: I originally got this idea from Dennis Lindley's wonderful article "The Analysis of Experimental Data: The Appreciation of Tea & Wine" in which he demonstrates why Bayesian methods are superior to classical methods.]

The back story is that Muriel Bristol visits Fisher one afternoon in the 1920's at Rothamsted Experimental Station for a cup of tea. When Fisher put the milk in last she complained saying that she could also tell whether the milk was poured first (or last) and that she preferred the former. To put this to the test he designed his classic tea experiment where Muriel is presented with a pair of tea cups and she must identify which one had the milk added first. This is repeated with six pairs of tea cups. Her choices are either Right (R) or Wrong (W) and her results are: RRRRRW.

Suppose that Muriel is actually just guessing and has no ability to discriminate whatsoever. This is called the Null Hypothesis. According to Fisher the purpose of the experiment is to discredit this null hypothesis. If Muriel is guessing she will identify the tea cup correctly with probability 0.5 on each turn and as they are independent the observed result has 0.5^6 = 0.016 (or 1/64). Fisher then argues that either:

(a) the null hypothesis (Muriel is guessing) is true and an event of small probability has occurred or, (b) the null hypothesis is false and Muriel has discriminatory powers.

The p-value (or probability value) is the probability of observing this outcome (RRRRRW) given the null hypothesis is true - it's the small probability referred to in (a), above. In this instance it's 0.016. Since events with small probabilities only occur rarely (by definition) situation (b) might be a more preferable explanation of what occurred than situation (a). When we reject the null hypothesis we're in fact accepting the opposite hypothesis which is we call the alternative hypothesis. In this example, Muriel has discriminatory powers is the alternative hypothesis.

An important consideration is what do we class as a "small" probability? What's the cutoff point at which we're willing to say that an event is unlikely? The standard benchmark is 5% (0.05) and this is called the significance level. When the p-value is smaller than the significance level we reject the null hypothesis as being false and accept our alternative hypothesis. It is common parlance to claim a result is "significant" when the p-value is smaller than the significance level i.e. when the probability of what we observed occurring given the null hypothesis is true is smaller than our cutoff point. It is important to be clear that using 5% is completely subjective (as is using the other common significance levels of 1% and 10%).

Fisher realised that this doesn't work; every possible outcome with one wrong pair was equally suggestive of discriminatory powers. The relevant probability for situation (a), above, is therefore 6(0.5)^6 = 0.094 (or 6/64) which now is not significant at a significance level of 5%. To overcome this Fisher argued that if 1 error in 6 is considered evidence of discriminatory powers then so is no errors i.e. outcomes that more strongly indicate discriminatory powers than the one observed should be included when calculating the p-value. This resulted in the following amendment to the reasoning, either:

(a) the null hypothesis (Muriel is guessing) is true and the probability of events as, or more, extreme than that observed is small, or (b) the null hypothesis is false and Muriel has discriminatory powers.

Back to our tea experiment and we find that the p-value under this set-up is 7(0.5)^6 = 0.109 which still is not significant at the 5% threshold.

I then get students to work with some other examples such as coin tossing to work out whether or not a coin is fair. This drills home the concepts of the null/alternative hypothesis, p-values and significance levels. We then move onto the case of a continuous variable and introduce the notion of a test-statistic. As we have already covered the normal distribution, standard normal distribution and the z-transformation in depth it's merely a matter of bolting together several concepts.

As well as calculating test-statistics, p-values and making a decision (significant/not significant) I get students to work through published papers in a fill in the missing blanks game.

Answer 4

A nice definition of p-value is "the probability of observing a test statistic at least as large as the one calculated assuming the null hypothesis is true".

The problem with that is that it requires an understanding of "test statistic" and "null hypothesis". But, that's easy to get across. If the null hypothesis is true, usually something like "parameter from population A is equal to parameter from population B", and you calculate statistics to estimate those parameters, what is the probability of seeing a test statistic that says, "they're this different"?

E.g., If the coin is fair, what is the probability I'd see 60 heads out of 100 tosses? That's testing the null hypothesis, "the coin is fair", or "p = .5" where p is the probability of heads.

The test statistic in that case would be the number of heads.

Now, I assume that what you're calling "t-value" is a generic "test statistic", not a value from a "t distribution". They're not the same thing, and the term "t-value" isn't (necessarily) widely used and could be confusing.

What you're calling "t-value" is probably what I'm calling "test statistic". In order to calculate a p-value (remember, it's just a probability) you need a distribution, and a value to plug into that distribution which will return a probability. Once you do that, the probability you return is your p-value. You can see that they are related because under the same distribution, different test-statistics are going to return different p-values. More extreme test-statistics will return lower p-values giving greater indication that the null hypothesis is false.

I've ignored the issue of one-sided and two-sided p-values here.

Answer 5

Imagine you have a bag containing 900 black marbles and 100 white, i.e. 10% of the marbles are white. Now imagine you take 1 marble out, look at it and record its colour, take out another, record its colour etc.. and do this 100 times. At the end of this process you will have a number for white marbles which, ideally, we would expect to be 10, i.e. 10% of 100, but in actual fact may be 8, or 13 or whatever simply due to randomness. If you repeat this 100 marble withdrawal experiment many, many times and then plot a histogram of the number of white marbles drawn per experiment, you'll find you will have a Bell Curve centred about 10.

This represents your 10% hypothesis: with any bag containing 1000 marbles of which 10% are white, if you randomly take out 100 marbles you will find 10 white marbles in the selection, give or take 4 or so. The p-value is all about this "give or take 4 or so." Let's say by referring to the Bell Curve created earlier you can determine that less than 5% of the time would you get 5 or fewer white marbles and another < 5% of the time accounts for 15 or more white marbles i.e. > 90% of the time your 100 marble selection will contain between 6 to 14 white marbles inclusive.

Now assuming someone plonks down a bag of 1000 marbles with an unknown number of white marbles in it, we have the tools to answer these questions

i) Are there fewer than 100 white marbles?

ii) Are there more than 100 white marbles?

iii) Does the bag contain 100 white marbles?

Simply take out 100 marbles from the bag and count how many of this sample are white.

a) If there are 6 to 14 whites in the sample you cannot reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 6 through 14 will be > 0.05.

b) If there are 5 or fewer whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 5 or fewer will be < 0.05. You would expect the bag to contain < 10% white marbles.

c) If there are 15 or more whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 15 or more will be < 0.05. You would expect the bag to contain > 10% white marbles.

In response to Baltimark's comment

Given the example above, there is an approximately:-

4.8% chance of getter 5 white balls or fewer

1.85% chance of 4 or fewer

0.55% chance of 3 or fewer

0.1% chance of 2 or fewer

6.25% chance of 15 or more

3.25% chance of 16 or more

1.5% chance of 17 or more

0.65% chance of 18 or more

0.25% chance of 19 or more

0.1% chance of 20 or more

0.05% chance of 21 or more

These numbers were estimated from an empirical distribution generated by a simple Monte Carlo routine run in R and the resultant quantiles of the sampling distribution.

For the purposes of answering the original question, suppose you draw 5 white balls, there is only an approximate 4.8% chance that if the 1000 marble bag really does contain 10% white balls you would pull out only 5 whites in a sample of 100. This equates to a p value < 0.05. You now have to choose between

i) There really are 10% white balls in the bag and I have just been "unlucky" to draw so few

or

ii) I have drawn so few white balls that there can't really be 10% white balls (reject the hypothesis of 10% white balls)

Answer 6

What the p-value doesn't tell you is how likely it is that the null hypothesis is true. Under the conventional (Fisher) significance testing framework we first compute the likelihood of observing the data assuming the null hypothesis is true, this is the p-value. It seems intuitively reasonable then to assume the null hypothesis is probably false if the data are sufficiently unlikely to be observed under the null hypothesis. This is entirely reasonable. Statisticians tranditionally use a threshold and "reject the null hypothesis at the 95% significance level" if (1 - p) > 0.95; however this is just a convention that has proven reasonable in practice - it doesn't mean that there is less than 5% probability that the null hypothesis is false (and therefore a 95% probability that the alternative hypothesis is true). One reason that we can't say this is that we have not looked at the alternative hypothesis yet.

Imaging a function f() that maps the p-value onto the probability that the alternative hypothesis is true. It would be reasonable to assert that this function is strictly decreasing (such that the more likely the observations under the null hypothesis, the less likely the alternative hypothesis is true), and that it gives values between 0 and 1 (as it gives an estimate of probability). However, that is all that we know about f(), so while there is a relationship between p and the probability that the alternative hypothesis is true, it is uncalibrated. This means we cannot use the p-value to make quantitative statements about the plausibility of the nulll and alternatve hypotheses.

Caveat lector: It isn't really within the frequentist framework to speak of the probability that a hypothesis is true, as it isn't a random variable - it is either true or it isn't. So where I have talked of the probability of the truth of a hypothesis I have implicitly moved to a Bayesian interpretation. It is incorrect to mix Bayesian and frequentist, however there is always a temptation to do so as what we really want is an quantative indication of the relative plausibility/probability of the hypotheses. But this is not what the p-value provides.

Answer 7

In statistics you can never say something is absolutely certain, so statisticians use another approach to gauge whether a hypothesis is true or not. They try to reject all the other hypotheses that are not supported by the data.

To do this, statistical tests have a null hypothesis and an alternate hypothesis. The p-value reported from a statistical test is the likelihood of the result given that the null hypothesis was correct. That's why we want small p-values. The smaller they are, the less likely the result would be if the null hypothesis was correct. If the p-value is small enough (ie,it is very unlikely for the result to have occurred if the null hypothesis was correct), then the null hypothesis is rejected.

In this fashion, null hypotheses can be formulated and subsequently rejected. If the null hypothesis is rejected, you accept the alternate hypothesis as the best explanation. Just remember though that the alternate hypothesis is never certain, since the null hypothesis could have, by chance, generated the results.

Answer 8

I find it helpful to follow a sequence in which you explain concepts in the following order: (1) The z score and proportions above and below the z score assuming a normal curve. (2) The notion of a sampling distribution, and the z score for a given sample mean when the population standard deviation is known (and thence the one sample z test) (3) The one-sample t-test and the likelihood of a sample mean when the population standard deviation is unknown (replete with stories about the secret identity of a certain industrial statistician and why Guinness is Good For Statistics). (4) The two-sample t-test and the sampling distribution of mean differences. The ease with which introductory students grasp the t-test has much to do with the groundwork that is laid in preparation for this topic.

/* instructor of terrified students mode off */

Answer 9

The p-value isnt as mysterious as most analysts make it out to be. It is a way of not having to calculate the confidence interval for a t-test but simply determining the confidence level with which null hypothesis can be rejected.

ILLUSTRATION. You run a test. The p-value comes up as 0.1866 for Q-variable, 0.0023 for R-variable. (These are expressed in %).

If you are testing at a 95% confidence level to reject the null hypo;

for Q: 100-18.66= 81.34%

for R: 100-0.23= 99.77%.

At a 95% confidence level, Q gives an 81.34% confidence to reject. This falls below 95% and is unacceptable. ACCEPT NULL.

R gives a 99.77% confidence to reject null. Clearly above the desired 95%. We thus reject the null.

I just illustrated the reading of the p-value through a 'reverse way' of measuring it up to the confidence level at which we reject the null hypo.