# What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It seems that students easily learn how to perform the calculations required by a given test but get hung up on interpreting the results. Many computerized tools report test results in terms of "p values" or "t values".

How would you explain the following points to college students taking their first course in statistics:

• What does a "p-value" mean in relation to the hypothesis being tested? Are there cases when one should be looking for a high p-value or a low p-value?

• What is the relationship between a p-value and a t-value?

• A fair bit of this is basically covered by the first sentence of the wikipedia article on p values, which correctly defines a p-value. If that's understood, much is made clear. – Glen_b -Reinstate Monica May 16 '13 at 7:50
• Just get the book: Statistics without Tears. It might save your sanity!! – user48700 Jun 20 '14 at 5:18
• @user48700 Could you summarize how Statistics Without Tears explains this? – Matt Krause Jun 20 '14 at 5:40
• Someone should draw a graph of p-value related questions over time and I bet we'll see the seasonality and correlation to academic calendars in colleges or Coursera data science classes – Aksakal Dec 29 '14 at 23:04
• In addition to other nice and relevant book recommendations in the answers and comments, I would like to suggest another book, appropriately called "What is a p-value anyway?". – Aleksandr Blekh Dec 29 '14 at 23:04

## Understanding $p$-value

Suppose, that you want to test the hypothesis that the average height of male students at your University is $5$ ft $7$ inches. You collect heights of $100$ students selected at random and compute the sample mean (say it turns out to be $5$ ft $9$ inches). Using an appropriate formula/statistical routine you compute the $p$-value for your hypothesis and say it turns out to be $0.06$.

In order to interpret $p=0.06$ appropriately, we should keep several things in mind:

1. The first step under classical hypothesis testing is the assumption that the hypothesis under consideration is true. (In our context, we assume that the true average height is $5$ ft $7$ inches.)

2. Imagine doing the following calculation: Compute the probability that the sample mean is greater than $5$ ft $9$ inches assuming that our hypothesis is in fact correct (see point 1).

In other words, we want to know $$\mathrm{P}(\mathrm{Sample\: mean} \ge 5 \:\mathrm{ft} \:9 \:\mathrm{inches} \:|\: \mathrm{True\: value} = 5 \:\mathrm{ft}\: 7\: \mathrm{inches}).$$

The calculation in step 2 is what is called the $p$-value. Therefore, a $p$-value of $0.06$ would mean that if we were to repeat our experiment many, many times (each time we select $100$ students at random and compute the sample mean) then $6$ times out of $100$ we can expect to see a sample mean greater than or equal to $5$ ft $9$ inches.

Given the above understanding, should we still retain our assumption that our hypothesis is true (see step 1)? Well, a $p=0.06$ indicates that one of two things have happened:

• (A) Either our hypothesis is correct and an extremely unlikely event has occurred (e.g., all $100$ students are student athletes)

or

• (B) Our assumption is incorrect and the sample we have obtained is not that unusual.

The traditional way to choose between (A) and (B) is to choose an arbitrary cut-off for $p$. We choose (A) if $p > 0.05$ and (B) if $p < 0.05$.

• Take your time! I won't be thinking about selecting a "Best Answer" for a week or so. – Sharpie Jul 19 '10 at 20:54
• Now that I've had a chance to come back and read the whole answer- a big +1 for the student height example. Very clear and well laid out. – Sharpie Jul 20 '10 at 0:12
• Nice work ... but we need to add (C) our model (embodied in the formula/statistical routine) is wrong. – Andrew Robinson Jul 20 '10 at 5:07
• A t-value (or any other test statistic) is mostly an intermediate step. It's basically some statistic that was proven, under some assumptions, to have a well-known distribution. Since we know the distribution of the test statistic under the null, we can then use standard tables (today mostly software) to derive a p-value. – Gala May 16 '13 at 8:39
• Isn't the p-value derived as a result of doing the chi-square test and then from the chi-square table? Am wondering how come the probability calculated above indicated the p-value itself?! – London guy Mar 10 '15 at 12:06

## A Dialog Between a Teacher and a Thoughtful Student

Humbly submitted in the belief that not enough crayons have been used so far in this thread. A brief illustrated synopsis appears at the end.

Student: What does a p-value mean? A lot of people seem to agree it's the chance we will "see a sample mean greater than or equal to" a statistic or it's "the probability of observing this outcome ... given the null hypothesis is true" or where "my sample's statistic fell on [a simulated] distribution" and even "the probability of observing a test statistic at least as large as the one calculated assuming the null hypothesis is true".

Teacher: Properly understood, all those statements are correct in many circumstances.

Student: I don't see how most of them are relevant. Didn't you teach us that we have to state a null hypothesis $H_0$ and an alternative hypothesis $H_A$? How are they involved in these ideas of "greater than or equal to" or "at least as large" or the very popular "more extreme"?

Teacher: Because it can seem complicated in general, would it help for us to explore a concrete example?

Student: Sure. But please make it a realistic but simple one if you can.

Teacher: This theory of hypothesis testing historically began with the need of astronomers to analyze observational errors, so how about starting there. I was going through some old documents one day where a scientist described his efforts to reduce the measurement error in his apparatus. He had taken a lot of measurements of a star in a known position and recorded their displacements ahead of or behind that position. To visualize those displacements, he drew a histogram that--when smoothed a little--looked like this one.

Student: I remember how histograms work: the vertical axis is labeled "Density" to remind me that the relative frequencies of the measurements are represented by area rather than height.

Teacher: That's right. An "unusual" or "extreme" value would be located in a region with pretty small area. Here's a crayon. Do you think you could color in a region whose area is just one-tenth the total?

Student: Sure; that's easy. [Colors in the figure.]

Teacher: Very good! That looks like about 10% of the area to me. Remember, though, that the only areas in the histogram that matter are those between vertical lines: they represent the chance or probability that the displacement would be located between those lines on the horizontal axis. That means you needed to color all the way down to the bottom and that would be over half the area, wouldn't it?

Student: Oh, I see. Let me try again. I'm going to want to color in where the curve is really low, won't I? It's lowest at the two ends. Do I have to color in just one area or would it be ok to break it into several parts?

Teacher: Using several parts is a smart idea. Where would they be?

Student (pointing): Here and here. Because this crayon isn't very sharp, I used a pen to show you the lines I'm using.

Teacher: Very nice! Let me tell you the rest of the story. The scientist made some improvements to his device and then he took additional measurements. He wrote that the displacement of the first one was only $0.1$, which he thought was a good sign, but being a careful scientist he proceeded to take more measurements as a check. Unfortunately, those other measurements are lost--the manuscript breaks off at this point--and all we have is that single number, $0.1$.

Student: That's too bad. But isn't that much better than the wide spread of displacements in your figure?

Teacher: That's the question I would like you to answer. To start with, what should we posit as $H_0$?

Student: Well, a sceptic would wonder whether the improvements made to the device had any effect at all. The burden of proof is on the scientist: he would want to show that the sceptic is wrong. That makes me think the null hypothesis is kind of bad for the scientist: it says that all the new measurements--including the value of $0.1$ we know about--ought to behave as described by the first histogram. Or maybe even worse than that: they might be even more spread out.

Teacher: Go on, you're doing well.

Student: And so the alternative is that the new measurements would be less spread out, right?

Teacher: Very good! Could you draw me a picture of what a histogram with less spread would look like? Here's another copy of the first histogram; you can draw on top of it as a reference.

Student (drawing): I'm using a pen to outline the new histogram and I'm coloring in the area beneath it. I have made it so most of the curve is close to zero on the horizontal axis and so most of its area is near a (horizontal) value of zero: that's what it means to be less spread out or more precise.

Teacher: That's a good start. But remember that a histogram showing chances should have a total area of $1$. The total area of the first histogram therefore is $1$. How much area is inside your new histogram?

Student: Less than half, I think. I see that's a problem, but I don't know how to fix it. What should I do?

Teacher: The trick is to make the new histogram higher than the old so that its total area is $1$. Here, I'll show you a computer-generated version to illustrate.

Student: I see: you stretched it out vertically so its shape didn't really change but now the red area and gray area (including the part under the red) are the same amounts.

Teacher: Right. You are looking at a picture of the null hypothesis (in blue, spread out) and part of the alternative hypothesis (in red, with less spread).

Student: What do you mean by "part" of the alternative? Isn't it just the alternative hypothesis?

Teacher: Statisticians and grammar don't seem to mix. :-) Seriously, what they mean by a "hypothesis" usually is a whole big set of possibilities. Here, the alternative (as you stated so well before) is that the measurements are "less spread out" than before. But how much less? There are many possibilities. Here, let me show you another. I drew it with yellow dashes. It's in between the previous two.

Student: I see: you can have different amounts of spread but you don't know in advance how much the spread will really be. But why did you make the funny shading in this picture?

Teacher: I wanted to highlight where and how the histograms differ. I shaded them in gray where the alternative histograms are lower than the null and in red where the alternatives are higher.

Student: Why would that matter?

Teacher: Do you remember how you colored the first histogram in both the tails? [Looking through the papers.] Ah, here it is. Let's color this picture in the same way.

Student: I remember: those are the extreme values. I found the places where the null density was as small as possible and colored in 10% of the area there.

Teacher: Tell me about the alternatives in those extreme areas.

Student: It's hard to see, because the crayon covered it up, but it looks like there's almost no chance for any alternative to be in the areas I colored. Their histograms are right down against value axis and there's no room for any area beneath them.

Teacher: Let's continue that thought. If I told you, hypothetically, that a measurement had a displacement of $-2$, and asked you to pick which of these three histograms was the one it most likely came from, which would it be?

Student: The first one--the blue one. It's the most spread out and it's the only one where $-2$ seems to have any chance of occurring.

Teacher: And what about the value of $0.1$ in the manuscript?

Student: Hmmm... that's a different story. All three histograms are pretty high above the ground at $0.1$.

Teacher: OK, fair enough. But suppose I told you the value was somewhere near $0.1$, like between $0$ and $0.2$. Does that help you read some probabilities off of these graphs?

Student: Sure, because I can use areas. I just have to estimate the areas underneath each curve between $0$ and $0.2$. But that looks pretty hard.

Teacher: You don't need to go that far. Can you just tell which area is the largest?

Student: The one beneath the tallest curve, of course. All three areas have the same base, so the taller the curve, the more area there is beneath it and the base. That means the tallest histogram--the one I drew, with the red dashes--is the likeliest one for a displacement of $0.1$. I think I see where you're going with this, but I'm a little concerned: don't I have to look at all the histograms for all the alternatives, not just the one or two shown here? How could I possibly do that?

Teacher: You're good at picking up patterns, so tell me: as the measurement apparatus is made more and more precise, what happens to its histogram?

Student: It gets narrower--oh, and it has to get taller, too, so its total area stays the same. That makes it pretty hard to compare the histograms. The alternative ones are all higher than the null right at $0$, that's obvious. But at other values sometimes the alternatives are higher and sometimes they are lower! For example, [pointing at a value near $3/4$], right here my red histogram is the lowest, the yellow histogram is the highest, and the original null histogram is between them. But over on the right the null is the highest.

Teacher: In general, comparing histograms is a complicated business. To help us do it, I have asked the computer to make another plot: it has divided each of the alternative histogram heights (or "densities") by the null histogram height, creating values known as "likelihood ratios." As a result, a value greater than $1$ means the alternative is more likely, while a value less than $1$ means the alternative is less likely. It has drawn yet one more alternative: it's more spread out than the other two, but still less spread out than the original apparatus was.

Teacher (continuing): Could you show me where the alternatives tend to be more likely than the null?

Student (coloring): Here in the middle, obviously. And because these are not histograms anymore, I guess we should be looking at heights rather than areas, so I'm just marking a range of values on the horizontal axis. But how do I know how much of the middle to color in? Where do I stop coloring?

Teacher: There's no firm rule. It all depends on how we plan to use our conclusions and how fierce the sceptics are. But sit back and think about what you have accomplished: you now realize that outcomes with large likelihood ratios are evidence for the alternative and outcomes with small likelihood ratios are evidence against the alternative. What I will ask you to do is to color in an area that, insofar as is possible, has a small chance of occurring under the null hypothesis and a relatively large chance of occurring under the alternatives. Going back to the first diagram you colored, way back at the start of our conversation, you colored in the two tails of the null because they were "extreme." Would they still do a good job?

Student: I don't think so. Even though they were pretty extreme and rare under the null hypothesis, they are practically impossible for any of the alternatives. If my new measurement were, say $3.0$, I think I would side with the sceptic and deny that any improvement had occurred, even though $3.0$ was an unusual outcome in any case. I want to change that coloring. Here--let me have another crayon.

Teacher: What does that represent?

Student: We started out with you asking me to draw in just 10% of the area under the original histogram--the one describing the null. So now I drew in 10% of the area where the alternatives seem more likely to be occurring. I think that when a new measurement is in that area, it's telling us we ought to believe the alternative.

Teacher: And how should the sceptic react to that?

Student: A sceptic never has to admit he's wrong, does he? But I think his faith should be a little shaken. After all, we arranged it so that although a measurement could be inside the area I just drew, it only has a 10% chance of being there when the null is true. And it has a larger chance of being there when the alternative is true. I just can't tell you how much larger that chance is, because it would depend on how much the scientist improved the apparatus. I just know it's larger. So the evidence would be against the sceptic.

Teacher: All right. Would you mind summarizing your understanding so that we're perfectly clear about what you have learned?

Student: I learned that to compare alternative hypotheses to null hypotheses, we should compare their histograms. We divide the densities of the alternatives by the density of the null: that's what you called the "likelihood ratio." To make a good test, I should pick a small number like 10% or whatever might be enough to shake a sceptic. Then I should find values where the likelihood ratio is as high as possible and color them in until 10% (or whatever) has been colored.

Teacher: And how would you use that coloring?

Student: As you reminded me earlier, the coloring has to be between vertical lines. Values (on the horizontal axis) that lie under the coloring are evidence against the null hypothesis. Other values--well, it's hard to say what they might mean without taking a more detailed look at all the histograms involved.

Teacher: Going back to the value of $0.1$ in the manuscript, what would you conclude?

Student: That's within the area I last colored, so I think the scientist probably was right and the apparatus really was improved.

Teacher: One last thing. Your conclusion was based on picking 10% as the criterion, or "size" of the test. Many people like to use 5% instead. Some prefer 1%. What could you tell them?

Student: I couldn't do all those tests at once! Well, maybe I could in a way. I can see that no matter what size the test should be, I ought to start coloring from $0$, which is in this sense the "most extreme" value, and work outwards in both directions from there. If I were to stop right at $0.1$--the value actually observed--I think I would have colored in an area somewhere between $0.05$ and $0.1$, say $0.08$. The 5% and 1% people could tell right away that I colored too much: if they wanted to color just 5% or 1%, they could, but they wouldn't get as far out as $0.1$. They wouldn't come to the same conclusion I did: they would say there's not enough evidence that a change actually occurred.

Teacher: You have just told me what all those quotations at the beginning really mean. It should be obvious from this example that they cannot possibly intend "more extreme" or "greater than or equal" or "at least as large" in the sense of having a bigger value or even having a value where the null density is small. They really mean these things in the sense of large likelihood ratios that you have described. By the way, the number around $0.08$ that you computed is called the "p-value." It can only properly be understood in the way you have described: with respect to an analysis of relative histogram heights--the likelihood ratios.

Student: Thank you. I'm not confident I fully understand all of this yet, but you have given me a lot to think about.

Teacher: If you would like to go further, take a look at the Neyman-Pearson Lemma. You are probably ready to understand it now.

## Synopsis

Many tests that are based on a single statistic like the one in the dialog will call it "$z$" or "$t$". These are ways of hinting what the null histogram looks like, but they are only hints: what we name this number doesn't really matter. The construction summarized by the student, as illustrated here, shows how it is related to the p-value. The p-value is the smallest test size that would cause an observation of $t=0.1$ to lead to a rejection of the null hypothesis.

In this figure, which is zoomed to show detail, the null hypothesis is plotted in solid blue and two typical alternatives are plotted with dashed lines. The region where those alternatives tend to be much larger than the null is shaded in. The shading starts where the relative likelihoods of the alternatives are greatest (at $0$). The shading stops when the observation $t=0.1$ is reached. The p-value is the area of the shaded region under the null histogram: it is the chance, assuming the null is true, of observing an outcome whose likelihood ratios tend to be large regardless of which alternative happens to be true. In particular, this construction depends intimately on the alternative hypothesis. It cannot be carried out without specifying the possible alternatives.

• This has excellently dealt with my comment on another answer, that none of the prior answers to this question had tackled, in generality, the commonly-heard "or more extreme" aspect of a p-value. (Though the "tea-testing" answer included a good specific example.) I particularly admire the way this example has been deliberately constructed to highlight that "more extreme" can mean quite the contrary of "bigger" or "further from zero". – Silverfish Jan 5 '15 at 20:47
• I wish teachers and textbooks didn't use the phrase "or more extreme", really. Two variants I have heard might be paraphrased as "more favourable towards $H_1$" or "more persuasive of $H_1$". In this instance, values nearer zero would indeed be more persuasive that the telescope has become more reliable, but it requires some linguistic acrobatics (plausibly argued, but potentially confusing) to describe them as "more extreme". – Silverfish Jan 5 '15 at 20:51
• Uniquely insightful as always, thank you for taking the time to write out those incredibly helpful answers. I really wonder why textbooks are never written in a way that offers anywhere near these levels of clarity and intuition. – jeremy radcliff Sep 20 '16 at 15:47
• I think a link to a definition of likelihood wrt this example could be beneficial – baxx Apr 27 '17 at 18:10
• It's dangerous to use sarcasm in a comment, @baxx, because there's not enough space allowed us to do so politely and elegantly. Therefore it's usually not a good idea to suppose a comment is being sarcastic unless it explicitly tells you so. Just assume comments are intended to help you. If you would simply follow the very first hit in the search I provided, I think your questions would be answered. – whuber Apr 27 '17 at 18:56

Before touching this topic, I always make sure that students are happy moving between percentages, decimals, odds and fractions. If they are not completely happy with this then they can get confused very quickly.

I like to explain hypothesis testing for the first time (and therefore p-values and test statistics) through Fisher's classic tea experiment. I have several reasons for this:

(i) I think working through an experiment and defining the terms as we go along makes more sense that just defining all of these terms to begin with. (ii) You don't need to rely explicitly on probability distributions, areas under the curve, etc to get over the key points of hypothesis testing. (iii) It explains this ridiculous notion of "as or more extreme than those observed" in a fairly sensible manner (iv) I find students like to understand the history, origins and back story of what they are studying as it makes it more real than some abstract theories. (v) It doesn't matter what discipline or subject the students come from, they can relate to the example of tea (N.B. Some international students have difficulty with this peculiarly British institution of tea with milk.)

[Note: I originally got this idea from Dennis Lindley's wonderful article "The Analysis of Experimental Data: The Appreciation of Tea & Wine" in which he demonstrates why Bayesian methods are superior to classical methods.]

The back story is that Muriel Bristol visits Fisher one afternoon in the 1920's at Rothamsted Experimental Station for a cup of tea. When Fisher put the milk in last she complained saying that she could also tell whether the milk was poured first (or last) and that she preferred the former. To put this to the test he designed his classic tea experiment where Muriel is presented with a pair of tea cups and she must identify which one had the milk added first. This is repeated with six pairs of tea cups. Her choices are either Right (R) or Wrong (W) and her results are: RRRRRW.

Suppose that Muriel is actually just guessing and has no ability to discriminate whatsoever. This is called the Null Hypothesis. According to Fisher the purpose of the experiment is to discredit this null hypothesis. If Muriel is guessing she will identify the tea cup correctly with probability 0.5 on each turn and as they are independent the observed result has 0.5$^6$ = 0.016 (or 1/64). Fisher then argues that either:

(a) the null hypothesis (Muriel is guessing) is true and an event of small probability has occurred or,

(b) the null hypothesis is false and Muriel has discriminatory powers.

The p-value (or probability value) is the probability of observing this outcome (RRRRRW) given the null hypothesis is true - it's the small probability referred to in (a), above. In this instance it's 0.016. Since events with small probabilities only occur rarely (by definition) situation (b) might be a more preferable explanation of what occurred than situation (a). When we reject the null hypothesis we're in fact accepting the opposite hypothesis which is we call the alternative hypothesis. In this example, Muriel has discriminatory powers is the alternative hypothesis.

An important consideration is what do we class as a "small" probability? What's the cutoff point at which we're willing to say that an event is unlikely? The standard benchmark is 5% (0.05) and this is called the significance level. When the p-value is smaller than the significance level we reject the null hypothesis as being false and accept our alternative hypothesis. It is common parlance to claim a result is "significant" when the p-value is smaller than the significance level i.e. when the probability of what we observed occurring given the null hypothesis is true is smaller than our cutoff point. It is important to be clear that using 5% is completely subjective (as is using the other common significance levels of 1% and 10%).

Fisher realised that this doesn't work; every possible outcome with one wrong pair was equally suggestive of discriminatory powers. The relevant probability for situation (a), above, is therefore 6(0.5)^6 = 0.094 (or 6/64) which now is not significant at a significance level of 5%. To overcome this Fisher argued that if 1 error in 6 is considered evidence of discriminatory powers then so is no errors i.e. outcomes that more strongly indicate discriminatory powers than the one observed should be included when calculating the p-value. This resulted in the following amendment to the reasoning, either:

(a) the null hypothesis (Muriel is guessing) is true and the probability of events as, or more, extreme than that observed is small, or

(b) the null hypothesis is false and Muriel has discriminatory powers.

Back to our tea experiment and we find that the p-value under this set-up is 7(0.5)^6 = 0.109 which still is not significant at the 5% threshold.

I then get students to work with some other examples such as coin tossing to work out whether or not a coin is fair. This drills home the concepts of the null/alternative hypothesis, p-values and significance levels. We then move onto the case of a continuous variable and introduce the notion of a test-statistic. As we have already covered the normal distribution, standard normal distribution and the z-transformation in depth it's merely a matter of bolting together several concepts.

As well as calculating test-statistics, p-values and making a decision (significant/not significant) I get students to work through published papers in a fill in the missing blanks game.

• I know I'm somewhat reviving a very old thread, but here it goes... I was really enjoying your answer, but I miss the t-value part in it :( Could you please use your given examples to talk about it? No one answered about the t-test part – Sosi Dec 4 '14 at 12:01
• @sosi It's probably because p-values are much more general than t-values. It's like asking a question about cars and then about the brakes on a Ford Fiesta. – conjectures Dec 29 '14 at 19:55
• The answer is very interesting (+1), but a few things are confused together at the end. 1. What does it mean for a $p$-value to be "significant at the 5% level"? Either the $p$-value is below 5%, or it is not. I don't see the point in using such an obscure sentence, leaving "significance" undefined. 2. What does it mean to "decide" wether or not a $p$-value is significant? It does not seem justified to bring in decision theory into the mix in this way (especially since Fisher was a strong opponent of the application of the Neyman-Pearson testing framework in the sciences). – Olivier Apr 2 '17 at 19:22

No amount of verbal explanation or calculations really helped me to understand at a gut level what p-values were, but it really snapped into focus for me once I took a course that involved simulation. That gave me the ability to actually see data generated by the null hypothesis and to plot the means/etc. of simulated samples, then look at where my sample's statistic fell on that distribution.

I think the key advantage to this is that it lets students forget about the math and the test statistic distributions for a minute and focus on the concepts at hand. Granted, it required that I learn how to simulate that stuff, which will cause problems for an entirely different set of students. But it worked for me, and I've used simulation countless times to help explain statistics to others with great success (e.g., "This is what your data looks like; this is what a Poisson distribution looks like overlaid. Are you SURE you want to do a Poisson regression?").

This doesn't exactly answer the questions you posed, but for me, at least, it made them trivial.

• I agree wholeheartedly about the use of simulation for explaining this. But just a small note on the example at the end: I find that people (not just students) do find it difficult to distinguish for any particular distributional assumption, e.g. the poisson, between being marginally poisson distributed and being conditionally poisson distributed. Since only the latter matters for a regression model, a bunch of dependent variable values that aren't poisson need not necessarily be any cause for concern. – conjugateprior Oct 30 '10 at 9:19
• I have to confess that I didn't know that. I've really appreciated your comments around this site over the past few days of your membership - I hope you'll stick around. – Matt Parker Oct 30 '10 at 18:29
• @MattParker do you know of any learning resources focussed towards the use of simulation to develop understanding? Or is it just a case of putting some python / R scripts together and running a bunch of tests? – baxx Apr 27 '17 at 13:39
• @baxx The [Seeing Theory website by Daniel Kunin](students.brown.edu/seeing-theory/) has some interesting tools for this, but it's still under construction. Otherwise, yeah, I've largely just experimented with R's built-in tools for simulation - using them to prove to myself how some method works, or to see what would happen if a predictor was replaced with a random variable, etc. Sorry, I wish I knew of better resources for this! – Matt Parker Apr 27 '17 at 19:48
• @MattParker cool thanks. Yeah - bit of a chicken and egg in that, to construct the experiments you (I assume?) need to at least get enough to write them. No worries though..... Just checked that site you linked, it's nice, thanks – baxx Apr 27 '17 at 20:13

A nice definition of p-value is "the probability of observing a test statistic at least as large as the one calculated assuming the null hypothesis is true".

The problem with that is that it requires an understanding of "test statistic" and "null hypothesis". But, that's easy to get across. If the null hypothesis is true, usually something like "parameter from population A is equal to parameter from population B", and you calculate statistics to estimate those parameters, what is the probability of seeing a test statistic that says, "they're this different"?

E.g., If the coin is fair, what is the probability I'd see 60 heads out of 100 tosses? That's testing the null hypothesis, "the coin is fair", or "p = .5" where p is the probability of heads.

The test statistic in that case would be the number of heads.

Now, I assume that what you're calling "t-value" is a generic "test statistic", not a value from a "t distribution". They're not the same thing, and the term "t-value" isn't (necessarily) widely used and could be confusing.

What you're calling "t-value" is probably what I'm calling "test statistic". In order to calculate a p-value (remember, it's just a probability) you need a distribution, and a value to plug into that distribution which will return a probability. Once you do that, the probability you return is your p-value. You can see that they are related because under the same distribution, different test-statistics are going to return different p-values. More extreme test-statistics will return lower p-values giving greater indication that the null hypothesis is false.

I've ignored the issue of one-sided and two-sided p-values here.

Imagine you have a bag containing 900 black marbles and 100 white, i.e. 10% of the marbles are white. Now imagine you take 1 marble out, look at it and record its colour, take out another, record its colour etc.. and do this 100 times. At the end of this process you will have a number for white marbles which, ideally, we would expect to be 10, i.e. 10% of 100, but in actual fact may be 8, or 13 or whatever simply due to randomness. If you repeat this 100 marble withdrawal experiment many, many times and then plot a histogram of the number of white marbles drawn per experiment, you'll find you will have a Bell Curve centred about 10.

This represents your 10% hypothesis: with any bag containing 1000 marbles of which 10% are white, if you randomly take out 100 marbles you will find 10 white marbles in the selection, give or take 4 or so. The p-value is all about this "give or take 4 or so." Let's say by referring to the Bell Curve created earlier you can determine that less than 5% of the time would you get 5 or fewer white marbles and another < 5% of the time accounts for 15 or more white marbles i.e. > 90% of the time your 100 marble selection will contain between 6 to 14 white marbles inclusive.

Now assuming someone plonks down a bag of 1000 marbles with an unknown number of white marbles in it, we have the tools to answer these questions

i) Are there fewer than 100 white marbles?

ii) Are there more than 100 white marbles?

iii) Does the bag contain 100 white marbles?

Simply take out 100 marbles from the bag and count how many of this sample are white.

a) If there are 6 to 14 whites in the sample you cannot reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 6 through 14 will be > 0.05.

b) If there are 5 or fewer whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 5 or fewer will be < 0.05. You would expect the bag to contain < 10% white marbles.

c) If there are 15 or more whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 15 or more will be < 0.05. You would expect the bag to contain > 10% white marbles.

In response to Baltimark's comment

Given the example above, there is an approximately:-

4.8% chance of getter 5 white balls or fewer

1.85% chance of 4 or fewer

0.55% chance of 3 or fewer

0.1% chance of 2 or fewer

6.25% chance of 15 or more

3.25% chance of 16 or more

1.5% chance of 17 or more

0.65% chance of 18 or more

0.25% chance of 19 or more

0.1% chance of 20 or more

0.05% chance of 21 or more

These numbers were estimated from an empirical distribution generated by a simple Monte Carlo routine run in R and the resultant quantiles of the sampling distribution.

For the purposes of answering the original question, suppose you draw 5 white balls, there is only an approximate 4.8% chance that if the 1000 marble bag really does contain 10% white balls you would pull out only 5 whites in a sample of 100. This equates to a p value < 0.05. You now have to choose between

i) There really are 10% white balls in the bag and I have just been "unlucky" to draw so few

or

ii) I have drawn so few white balls that there can't really be 10% white balls (reject the hypothesis of 10% white balls)

• First of all, this is just a big example and doesn't really explain explain the concept of p-value and test-statistic. Second, you're just claiming that if you get fewer than 5 or more than 15 white marbles, you reject the null hypothesis. What's your distribution that you're calculating those probabilities from? This can be approximated with a normal dist. centered at 10, with a standard deviation of 3. Your rejection criteria is not nearly strict enough. – Baltimark Jul 20 '10 at 15:21
• I would agree that this is just an example, and I it is true I just picked the numbers 5 and 15 out of the air for illustrative purposes. When I have time I will post a second answer, which I hope will be more complete. – babelproofreader Jul 20 '10 at 22:00

What the p-value doesn't tell you is how likely it is that the null hypothesis is true. Under the conventional (Fisher) significance testing framework we first compute the likelihood of observing the data assuming the null hypothesis is true, this is the p-value. It seems intuitively reasonable then to assume the null hypothesis is probably false if the data are sufficiently unlikely to be observed under the null hypothesis. This is entirely reasonable. Statisticians tranditionally use a threshold and "reject the null hypothesis at the 95% significance level" if (1 - p) > 0.95; however this is just a convention that has proven reasonable in practice - it doesn't mean that there is less than 5% probability that the null hypothesis is false (and therefore a 95% probability that the alternative hypothesis is true). One reason that we can't say this is that we have not looked at the alternative hypothesis yet.

Imaging a function f() that maps the p-value onto the probability that the alternative hypothesis is true. It would be reasonable to assert that this function is strictly decreasing (such that the more likely the observations under the null hypothesis, the less likely the alternative hypothesis is true), and that it gives values between 0 and 1 (as it gives an estimate of probability). However, that is all that we know about f(), so while there is a relationship between p and the probability that the alternative hypothesis is true, it is uncalibrated. This means we cannot use the p-value to make quantitative statements about the plausibility of the nulll and alternatve hypotheses.

Caveat lector: It isn't really within the frequentist framework to speak of the probability that a hypothesis is true, as it isn't a random variable - it is either true or it isn't. So where I have talked of the probability of the truth of a hypothesis I have implicitly moved to a Bayesian interpretation. It is incorrect to mix Bayesian and frequentist, however there is always a temptation to do so as what we really want is an quantative indication of the relative plausibility/probability of the hypotheses. But this is not what the p-value provides.

In statistics you can never say something is absolutely certain, so statisticians use another approach to gauge whether a hypothesis is true or not. They try to reject all the other hypotheses that are not supported by the data.

To do this, statistical tests have a null hypothesis and an alternate hypothesis. The p-value reported from a statistical test is the likelihood of the result given that the null hypothesis was correct. That's why we want small p-values. The smaller they are, the less likely the result would be if the null hypothesis was correct. If the p-value is small enough (ie,it is very unlikely for the result to have occurred if the null hypothesis was correct), then the null hypothesis is rejected.

In this fashion, null hypotheses can be formulated and subsequently rejected. If the null hypothesis is rejected, you accept the alternate hypothesis as the best explanation. Just remember though that the alternate hypothesis is never certain, since the null hypothesis could have, by chance, generated the results.

• a p-value is the likelihood of a result as or more "extreme" than the result given, not of the actual result. p-value is $Pr(T\geq t|H_0)$ and not $Pr(T=t|H_0)$ (T is test statistic, and t is its observed value). – probabilityislogic Jul 3 '11 at 1:05

I am bit diffident to revive the old topic, but I jumped from here, so I post this as a response to the question in the link.

The p-value is a concrete term, there should be no room for misunderstanding. But, it is somehow mystical that colloquial translations of the definition of p-value leads to many different misinterpretations. I think the root of the problem is in the use of the phrases "at least as adverse to null hypothesis" or "at least as extreme as the one in your sample data" etc.

For instance, Wikipedia says

...the p-value is the probability of obtaining the observed sample results (or a more extreme result) when the null hypothesis is actually true.

Meaning of $p$-value is blurred when people first stumble upon "(or a more extreme result)" and start thinking "more extreeeme?".

I think it is better to leave the "more extreme result" to something like indirect speech act. So, my take is

The p-value is the probability of seeing what you see in a "imaginary world" where the null hypothesis is true.

To make the idea concrete, suppose you have sample x consisting of 10 observations and you hypothesize that the population mean is $\mu_0=20$. So, in your hypothesized world, population distribution is $N(20,1)$.

x
#[1] 20.82600 19.30229 18.74753 18.99071 20.14312 16.76647
#[7] 18.94962 17.99331 19.22598 18.68633


You compute t-stat as $t_0=\sqrt{n}\frac{\bar{X}-\mu_0}{s}$, and find out that

sqrt(10) * (mean(x) - 20) / sd(x)
#-2.974405


So, what is the probability of observing $|t_0|$ as large as 2.97 ( "more extreme" comes here) in the imaginary world? In the imaginary world $t_0\sim t(9)$, thus, the p-value must be $$p-value=Pr(|t_0|\geq 2.97)= 0.01559054$$

2*(1 - pt(2.974405, 9))
#[1] 0.01559054


Since p-value is small, it is very unlikely that the sample x would have been drawn in the hypothesized world. Therefore, we conclude that it is very unlikely that the hypothesized world was in fact the actual world.

• +1, but when you write "probability of seeing what you see" and omit the "more extreme" part, this sentence becomes strictly speaking false (and potentially misleading, even if perhaps less confusing). It is not the probability of seeing what you see (this is usually zero). It is the probability of seeing what you see "or more extreme". Even though this might be a confusing bit for many, it is still crucial (and one can argue endlessly about the degree of subjectivity that hides behind this "more extreme" wording). – amoeba says Reinstate Monica Dec 30 '14 at 9:30
• @amoeba I thought, when adequate example supplied, it could serve as a proxy for "obtaining the observed sample results (or a more extreme result)". Maybe, better wording is needed. – Khashaa Dec 30 '14 at 10:09
• I was going to make the same observation as @amoeba; the "or more extreme" part is handled well by example in the student heights and tea party answers, but I don't think any answers in this thread have hit upon a clear general explanation of it, particularly one which covers different alternative hypotheses. I do agree with this answer suggesting that the "or more extreme" part is a conceptual sticking point for many students. – Silverfish Dec 30 '14 at 13:43
• @Silverfish: and not only students. How many Bayesian-vs-frequentists rants have I read that discuss the subjectivity/objectivity issue of this "more extreme" bit! – amoeba says Reinstate Monica Dec 30 '14 at 13:46
• @Silver I agree with your criticism and have posted an answer attempting to address it. "Or more extreme" is the very crux of the matter. – whuber Dec 30 '14 at 21:53

I find it helpful to follow a sequence in which you explain concepts in the following order: (1) The z score and proportions above and below the z score assuming a normal curve. (2) The notion of a sampling distribution, and the z score for a given sample mean when the population standard deviation is known (and thence the one sample z test) (3) The one-sample t-test and the likelihood of a sample mean when the population standard deviation is unknown (replete with stories about the secret identity of a certain industrial statistician and why Guinness is Good For Statistics). (4) The two-sample t-test and the sampling distribution of mean differences. The ease with which introductory students grasp the t-test has much to do with the groundwork that is laid in preparation for this topic.

/* instructor of terrified students mode off */

I have also found simulations to be a useful in teaching.

Here is a simulation for the arguably most basic case in which we sample $n$ times from $N(\mu,1)$ (hence, $\sigma^2=1$ is known for simplicity) and test $H_0:\mu=\mu_0$ against a left-sided alternative.

Then, the $t$-statistic $\text{tstat}:=\sqrt{n}(\bar{X}-\mu_0)$ is $N(0,1)$ under $H_0$, such that the $p$-value is simply $\Phi(\text{tstat})$ or pnorm(tstat) in R.

In the simulation, it is the fraction of times that data generated under the null $N(\mu_0,1)$ (here, $\mu_0=2$) yields sample means stored in nullMeans that are less (i.e., more extreme'' in this left-sided test) than the one calculated from the observed data.

# p value
set.seed(1)
reps <- 1000
n <- 100
mu <- 1.85 # true value
mu_0 <- 2 # null value
xaxis <- seq(-3, 3, length = 100)

X <- rnorm(n,mu)

nullMeans <- counter <- rep(NA,reps)

yvals <- jitter(rep(0,reps),2)

for (i in 1:reps)
{
tstat <- sqrt(n)*(mean(X)-mu_0) # test statistic, N(0,1) under the given assumptions

par(mfrow=c(1,3))
plot(xaxis,dnorm(xaxis),ylab="null distribution",xlab="possible test statistics",type="l")
points(tstat,0,cex=2,col="salmon",pch=21,bg="salmon")

X_null <- rnorm(n,mu_0) # generate data under H_0
nullMeans[i] <- mean(X_null)

plot(nullMeans[1:i],yvals[1:i],col="blue",pch=21,xlab="actual means and those generated under the null",ylab="", yaxt='n',ylim=c(-1,1),xlim=c(1.5,2.5))
abline(v=mu_0,lty=2)
points(mean(X),0,cex=4,col="salmon",pch=21,bg="salmon")

# counts 1 if sample generated under H_0 is more extreme:
counter[i] <- (nullMeans[i] < mean(X)) # i.e. we test against H_1: mu < mu_0
barplot(table(counter[1:i])/i,col=c("green","red"),xlab="more extreme mean under the null than the mean actually observed")

if(i<10) locator(1)
}
mean(counter)
pnorm(tstat)


What does a "p-value" mean in relation to the hypothesis being tested?

In an ontological sense (what is truth?), it means nothing. Any hypothesis testing is based on untested assumptions. This are normally part of the test itself, but are also part of whatever model you are using (e.g. in a regression model). Since we are merely assuming these, we cannot know if the reason why the p-value is below our threshold is because the null is false. It is a non sequitur to deduce unconditionally that because of a low p-value we must reject the null. For instance, something in the model could be wrong.

In an epistemological sense (what can we learn?), it means something. You gain knowledge conditional on the untested premises being true. Since (at least until now) we cannot prove every edifice of reality, all our knowledge will be necessarily conditional. We will never get to the "truth".

I think that examples involving marbles or coins or height-measuring can be fine for practicing the math, but they aren't good for building intuition. College students like to question society, right? How about using a political example?

Say a political candidate ran a campaign promising that some policy will help the economy. She was elected, she got the policy enacted, and 2 years later, the economy is booming. She's up for re-election, and claims that her policy is the reason for everyone's prosperity. Should you re-elect her?

The thoughtful citizen should say "well, it's true that the economy is doing well, but can we really attribute that to your policy?" To truly answer this, we must consider the question "would the economy have done well in the last 2 years without it?" If the answer is yes (e.g. the economy is booming because of some new unrelated technological development) then we reject the politician's explanation of the data.

That is, to examine one hypothesis (policy helped the economy), we must build a model of the world where that hypothesis is null (the policy was never enacted). We then make a prediction under that model. We call the probability of observing this data in that alternate world the p-value. If the p-value is too high, then we aren't convinced by the hypothesis--the policy made no difference. If the p-value is low then we trust the hypothesis--the policy was essential.

• I disagree with the p being defined as "We call the probability of observing this data in that alternate world the p-value" and also the strength of the conclusion being drawn (especially failure to reject the null). – Silverfish Dec 30 '14 at 13:46
• @Silverfish Could you elaborate? Probably it would be more correct to call the p-value the probability of making that observation OR a more extreme observation. But it sounds like you have a deeper criticism. – cgreen Jan 3 '15 at 19:45
• Since the original question is asking what a p-value is, I thought that getting that definition across clearly was important. Just saying "more extreme" isn't in itself very helpful without explaining what "more extreme" might mean - that's a weakness of most answers in this thread I think. Only whuber's answer and the "tea test" one seem to really explain why the "more extreme" matters too. – Silverfish Jan 3 '15 at 23:22
• I also felt your conclusions are phrased too strongly. If we reject the null, we have significant evidence against it, but don't know that it's false. When we fail to reject the null, that certainly doesn't mean the null is true (though it may well be). As a more general comment I have the feeling the test you're describing, in quite abstract terms, is not likely to be clear to a learner who is just learning how to perform a test. The lack of a clearly defined test statistic doesn't sit well with the original question asking how to interpret t-statistic too. – Silverfish Jan 3 '15 at 23:33
• A feature of this answer I like a lot is the clear explanation that p-values are calculated using a null model, even if we don't (subjectively) believe the null model is actually true. I think the fact test statistics are calculated under a model is a key point that many students struggle with. – Silverfish Jan 3 '15 at 23:37

I have yet to prove the following argument so it might contain errors, but I really want to throw in my two cents (Hopefully, I'll update it with a rigorous proof soon). Another way of looking at the $$p$$-value is

$$p$$-value - A statistic $$X$$ such that $$\forall 0 \le c \le 1, F_{X|H_0}(\inf\{x: F_{X|H_0}(x) \ge c\}) = c$$ where $$F_{X|H_0}$$ is the distribution function of $$X$$ under $$H_0$$.

Specifically, if $$X$$ has a continuous distribution and you're not using approximation, then

1. Every $$p$$-value is a statistic with a uniform distribution on $$[0, 1]$$, and
2. Every statistic with a uniform distribution on $$[0, 1]$$ is a $$p$$-value.

You may consider this a generalized description of the $$p$$-values.

• This definition makes sense only for discrete distributions (and then is not correct), because the second appearance of "$P$" makes it clear it refers to probabilities, not probability densities. Moreover, there are extremely few distributions (if any) which have the stated property, suggesting that there must be typographical errors in the statement. As far as your subsequent claims go, (1) is ideally true but (2) is not, unless you allow the null hypothesis to depend on the statistic! – whuber Jun 5 at 12:09
• @whuber Thanks for the input. I have edited the definition, and it should make more sense now! – nalzok Jun 5 at 15:47
• It does make sense, thank you: if I'm reading it correctly, it asserts the null distribution of $X$ is uniform on $[0,1].$ However, that captures only part of the properties of p-values; it does not characterize p-values; and it says nothing about what they mean or how to interpret them. Consider studying some of the other answers in this thread for information on what is missing. – whuber Jun 5 at 15:50
• Here is an example that you might find interesting. The distribution family is Uniform$(\theta,\theta+1)$ for $\theta\in\mathbb{R},$ the null hypothesis is $\theta=0,$ and the alternative is its complement. Consider a random sample $\mathbf{X}=(X_1,\ldots,X_n).$ Define the statistic $X(\mathbf{X}) = X_1.$ Obviously this has a uniform distribution on $[0,1]$ under $H_0:$ but in what sense is it a p-value? What is the corresponding hypothesis test? Suppose we take a sample of size $n=1$ and observe the value $X_1=-2:$ are you claiming the p-value is $-2$?? – whuber Jun 5 at 15:59

The p-value isnt as mysterious as most analysts make it out to be. It is a way of not having to calculate the confidence interval for a t-test but simply determining the confidence level with which null hypothesis can be rejected.

ILLUSTRATION. You run a test. The p-value comes up as 0.1866 for Q-variable, 0.0023 for R-variable. (These are expressed in %).

If you are testing at a 95% confidence level to reject the null hypo;

for Q: 100-18.66= 81.34%

for R: 100-0.23= 99.77%.

At a 95% confidence level, Q gives an 81.34% confidence to reject. This falls below 95% and is unacceptable. ACCEPT NULL.

R gives a 99.77% confidence to reject null. Clearly above the desired 95%. We thus reject the null.

I just illustrated the reading of the p-value through a 'reverse way' of measuring it up to the confidence level at which we reject the null hypo.

• Welcome to the site. What do you mean by $Q$-variable and $R$-variable? Please clarify. Also, use of the phrase "accept null" is usually considered quite undesirable, even misleading. – cardinal Jan 8 '12 at 3:46
• @cardinal points out an important point. You're not going to accept the null. – Patrick Coulombe Oct 13 '13 at 22:22

******p value in testing of hypothesis measures the sensitivity of the test .The lower the p value the greater is the sensitivity. if significance level is set at 0.05 the p value of 0.0001 indicates a high probability of the test results being correct******

• -1 This is clearly wrong. You may want to read the higher voted answers first. – Momo Jul 19 '14 at 10:04