What is the meaning of p values and t values in statistical tests?

Question

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It seems that students easily learn how to perform the calculations required by a given test but get hung up on interpreting the results. Many computerized tools report test results in terms of "p values" or "t values".

How would you explain the following points to college students taking their first course in statistics:

• What does a "p-value" mean in relation to the hypothesis being tested? Are there cases when one should be looking for a high p-value or a low p-value?

• What is the relationship between a p-value and a t-value?

Answer 1

Understanding $p$-value

Suppose, that you want to test the hypothesis that the average height of male students at your University is $5$ ft $7$ inches. You collect heights of $100$ students selected at random and compute the sample mean (say it turns out to be $5$ ft $9$ inches). Using an appropriate formula/statistical routine you compute the $p$-value for your hypothesis and say it turns out to be $0.06$.

In order to interpret $p=0.06$ appropriately, we should keep several things in mind:

1. The first step under classical hypothesis testing is the assumption that the hypothesis under consideration is true. (In our context, we assume that the true average height is $5$ ft $7$ inches.)

2. Imagine doing the following calculation: Compute the probability that the sample mean is greater than $5$ ft $9$ inches assuming that our hypothesis is in fact correct (see point 1).

In other words, we want to know $$\mathrm{P}(\mathrm{Sample\: mean} \ge 5 \:\mathrm{ft} \:9 \:\mathrm{inches} \:|\: \mathrm{True\: value} = 5 \:\mathrm{ft}\: 7\: \mathrm{inches}).$$

The calculation in step 2 is what is called the $p$-value. Therefore, a $p$-value of $0.06$ would mean that if we were to repeat our experiment many, many times (each time we select $100$ students at random and compute the sample mean) then $6$ times out of $100$ we can expect to see a sample mean greater than or equal to $5$ ft $9$ inches.

Given the above understanding, should we still retain our assumption that our hypothesis is true (see step 1)? Well, a $p=0.06$ indicates that one of two things have happened:

• (A) Either our hypothesis is correct and an extremely unlikely event has occurred (e.g., all $100$ students are student athletes)

or

• (B) Our assumption is incorrect and the sample we have obtained is not that unusual.

The traditional way to choose between (A) and (B) is to choose an arbitrary cut-off for $p$. We choose (A) if $p > 0.05$ and (B) if $p < 0.05$.

Answer 2

A Dialog Between a Teacher and a Thoughtful Student

Humbly submitted in the belief that not enough crayons have been used so far in this thread. A brief illustrated synopsis appears at the end. A practical, real-world example is worked out (with code) at https://stats.stackexchange.com/a/131489/919.

---

Student: What does a p-value mean? A lot of people seem to agree it's the chance we will "see a sample mean greater than or equal to" a statistic or it's "the probability of observing this outcome ... given the null hypothesis is true" or where "my sample's statistic fell on [a simulated] distribution" and even "the probability of observing a test statistic at least as large as the one calculated assuming the null hypothesis is true".

Teacher: Properly understood, all those statements are correct in many circumstances.

Student: I don't see how most of them are relevant. Didn't you teach us that we have to state a null hypothesis $H_0$ and an alternative hypothesis $H_A$? How are they involved in these ideas of "greater than or equal to" or "at least as large" or the very popular "more extreme"?

Teacher: Because it can seem complicated in general, would it help for us to explore a concrete example?

Student: Sure. But please make it a realistic but simple one if you can.

Teacher: This theory of hypothesis testing historically began with the need of astronomers to analyze observational errors, so how about starting there. I was going through some old documents one day where a scientist described his efforts to reduce the measurement error in his apparatus. He had taken a lot of measurements of a star in a known position and recorded their displacements ahead of or behind that position. To visualize those displacements, he drew a histogram that--when smoothed a little--looked like this one.

Student: I remember how histograms work: the vertical axis is labeled "Density" to remind me that the relative frequencies of the measurements are represented by area rather than height.

Teacher: That's right. An "unusual" or "extreme" value would be located in a region with pretty small area. Here's a crayon. Do you think you could color in a region whose area is just one-tenth the total?

Student: Sure; that's easy. [Colors in the figure.]

Teacher: Very good! That looks like about 10% of the area to me. Remember, though, that the only areas in the histogram that matter are those between vertical lines: they represent the chance or probability that the displacement would be located between those lines on the horizontal axis. That means you needed to color all the way down to the bottom and that would be over half the area, wouldn't it?

Student: Oh, I see. Let me try again. I'm going to want to color in where the curve is really low, won't I? It's lowest at the two ends. Do I have to color in just one area or would it be ok to break it into several parts?

Teacher: Using several parts is a smart idea. Where would they be?

Student (pointing): Here and here. Because this crayon isn't very sharp, I used a pen to show you the lines I'm using.

Teacher: Very nice! Let me tell you the rest of the story. The scientist made some improvements to his device and then he took additional measurements. He wrote that the displacement of the first one was only $0.1$, which he thought was a good sign, but being a careful scientist he proceeded to take more measurements as a check. Unfortunately, those other measurements are lost--the manuscript breaks off at this point--and all we have is that single number, $0.1$.

Student: That's too bad. But isn't that much better than the wide spread of displacements in your figure?

Teacher: That's the question I would like you to answer. To start with, what should we posit as $H_0$?

Student: Well, a sceptic would wonder whether the improvements made to the device had any effect at all. The burden of proof is on the scientist: he would want to show that the sceptic is wrong. That makes me think the null hypothesis is kind of bad for the scientist: it says that all the new measurements--including the value of $0.1$ we know about--ought to behave as described by the first histogram. Or maybe even worse than that: they might be even more spread out.

Teacher: Go on, you're doing well.

Student: And so the alternative is that the new measurements would be less spread out, right?

Teacher: Very good! Could you draw me a picture of what a histogram with less spread would look like? Here's another copy of the first histogram; you can draw on top of it as a reference.

Student (drawing): I'm using a pen to outline the new histogram and I'm coloring in the area beneath it. I have made it so most of the curve is close to zero on the horizontal axis and so most of its area is near a (horizontal) value of zero: that's what it means to be less spread out or more precise.

Teacher: That's a good start. But remember that a histogram showing chances should have a total area of $1$. The total area of the first histogram therefore is $1$. How much area is inside your new histogram?

Student: Less than half, I think. I see that's a problem, but I don't know how to fix it. What should I do?

Teacher: The trick is to make the new histogram higher than the old so that its total area is $1$. Here, I'll show you a computer-generated version to illustrate.

Student: I see: you stretched it out vertically so its shape didn't really change but now the red area and gray area (including the part under the red) are the same amounts.

Teacher: Right. You are looking at a picture of the null hypothesis (in blue, spread out) and part of the alternative hypothesis (in red, with less spread).

Student: What do you mean by "part" of the alternative? Isn't it just the alternative hypothesis?

Teacher: Statisticians and grammar don't seem to mix. :-) Seriously, what they mean by a "hypothesis" usually is a whole big set of possibilities. Here, the alternative (as you stated so well before) is that the measurements are "less spread out" than before. But how much less? There are many possibilities. Here, let me show you another. I drew it with yellow dashes. It's in between the previous two.

Student: I see: you can have different amounts of spread but you don't know in advance how much the spread will really be. But why did you make the funny shading in this picture?

Teacher: I wanted to highlight where and how the histograms differ. I shaded them in gray where the alternative histograms are lower than the null and in red where the alternatives are higher.

Student: Why would that matter?

Teacher: Do you remember how you colored the first histogram in both the tails? [Looking through the papers.] Ah, here it is. Let's color this picture in the same way.

Student: I remember: those are the extreme values. I found the places where the null density was as small as possible and colored in 10% of the area there.

Teacher: Tell me about the alternatives in those extreme areas.

Student: It's hard to see, because the crayon covered it up, but it looks like there's almost no chance for any alternative to be in the areas I colored. Their histograms are right down against value axis and there's no room for any area beneath them.

Teacher: Let's continue that thought. If I told you, hypothetically, that a measurement had a displacement of $-2$, and asked you to pick which of these three histograms was the one it most likely came from, which would it be?

Student: The first one--the blue one. It's the most spread out and it's the only one where $-2$ seems to have any chance of occurring.

Teacher: And what about the value of $0.1$ in the manuscript?

Student: Hmmm... that's a different story. All three histograms are pretty high above the ground at $0.1$.

Teacher: OK, fair enough. But suppose I told you the value was somewhere near $0.1$, like between $0$ and $0.2$. Does that help you read some probabilities off of these graphs?

Student: Sure, because I can use areas. I just have to estimate the areas underneath each curve between $0$ and $0.2$. But that looks pretty hard.

Teacher: You don't need to go that far. Can you just tell which area is the largest?

Student: The one beneath the tallest curve, of course. All three areas have the same base, so the taller the curve, the more area there is beneath it and the base. That means the tallest histogram--the one I drew, with the red dashes--is the likeliest one for a displacement of $0.1$. I think I see where you're going with this, but I'm a little concerned: don't I have to look at all the histograms for all the alternatives, not just the one or two shown here? How could I possibly do that?

Teacher: You're good at picking up patterns, so tell me: as the measurement apparatus is made more and more precise, what happens to its histogram?

Student: It gets narrower--oh, and it has to get taller, too, so its total area stays the same. That makes it pretty hard to compare the histograms. The alternative ones are all higher than the null right at $0$, that's obvious. But at other values sometimes the alternatives are higher and sometimes they are lower! For example, [pointing at a value near $3/4$], right here my red histogram is the lowest, the yellow histogram is the highest, and the original null histogram is between them. But over on the right the null is the highest.

Teacher: In general, comparing histograms is a complicated business. To help us do it, I have asked the computer to make another plot: it has divided each of the alternative histogram heights (or "densities") by the null histogram height, creating values known as "likelihood ratios." As a result, a value greater than $1$ means the alternative is more likely, while a value less than $1$ means the alternative is less likely. It has drawn yet one more alternative: it's more spread out than the other two, but still less spread out than the original apparatus was.

Teacher (continuing): Could you show me where the alternatives tend to be more likely than the null?

Student (coloring): Here in the middle, obviously. And because these are not histograms anymore, I guess we should be looking at heights rather than areas, so I'm just marking a range of values on the horizontal axis. But how do I know how much of the middle to color in? Where do I stop coloring?

Teacher: There's no firm rule. It all depends on how we plan to use our conclusions and how fierce the sceptics are. But sit back and think about what you have accomplished: you now realize that outcomes with large likelihood ratios are evidence for the alternative and outcomes with small likelihood ratios are evidence against the alternative. What I will ask you to do is to color in an area that, insofar as is possible, has a small chance of occurring under the null hypothesis and a relatively large chance of occurring under the alternatives. Going back to the first diagram you colored, way back at the start of our conversation, you colored in the two tails of the null because they were "extreme." Would they still do a good job?

Student: I don't think so. Even though they were pretty extreme and rare under the null hypothesis, they are practically impossible for any of the alternatives. If my new measurement were, say $3.0$, I think I would side with the sceptic and deny that any improvement had occurred, even though $3.0$ was an unusual outcome in any case. I want to change that coloring. Here--let me have another crayon.

Teacher: What does that represent?

Student: We started out with you asking me to draw in just 10% of the area under the original histogram--the one describing the null. So now I drew in 10% of the area where the alternatives seem more likely to be occurring. I think that when a new measurement is in that area, it's telling us we ought to believe the alternative.

Teacher: And how should the sceptic react to that?

Student: A sceptic never has to admit he's wrong, does he? But I think his faith should be a little shaken. After all, we arranged it so that although a measurement could be inside the area I just drew, it only has a 10% chance of being there when the null is true. And it has a larger chance of being there when the alternative is true. I just can't tell you how much larger that chance is, because it would depend on how much the scientist improved the apparatus. I just know it's larger. So the evidence would be against the sceptic.

Teacher: All right. Would you mind summarizing your understanding so that we're perfectly clear about what you have learned?

Student: I learned that to compare alternative hypotheses to null hypotheses, we should compare their histograms. We divide the densities of the alternatives by the density of the null: that's what you called the "likelihood ratio." To make a good test, I should pick a small number like 10% or whatever might be enough to shake a sceptic. Then I should find values where the likelihood ratio is as high as possible and color them in until 10% (or whatever) has been colored.

Teacher: And how would you use that coloring?

Student: As you reminded me earlier, the coloring has to be between vertical lines. Values (on the horizontal axis) that lie under the coloring are evidence against the null hypothesis. Other values--well, it's hard to say what they might mean without taking a more detailed look at all the histograms involved.

Teacher: Going back to the value of $0.1$ in the manuscript, what would you conclude?

Student: That's within the area I last colored, so I think the scientist probably was right and the apparatus really was improved.

Teacher: One last thing. Your conclusion was based on picking 10% as the criterion, or "size" of the test. Many people like to use 5% instead. Some prefer 1%. What could you tell them?

Student: I couldn't do all those tests at once! Well, maybe I could in a way. I can see that no matter what size the test should be, I ought to start coloring from $0$, which is in this sense the "most extreme" value, and work outwards in both directions from there. If I were to stop right at $0.1$--the value actually observed--I think I would have colored in an area somewhere between $0.05$ and $0.1$, say $0.08$. The 5% and 1% people could tell right away that I colored too much: if they wanted to color just 5% or 1%, they could, but they wouldn't get as far out as $0.1$. They wouldn't come to the same conclusion I did: they would say there's not enough evidence that a change actually occurred.

Teacher: You have just told me what all those quotations at the beginning really mean. It should be obvious from this example that they cannot possibly intend "more extreme" or "greater than or equal" or "at least as large" in the sense of having a bigger value or even having a value where the null density is small. They really mean these things in the sense of large likelihood ratios that you have described. By the way, the number around $0.08$ that you computed is called the "p-value." It can only properly be understood in the way you have described: with respect to an analysis of relative histogram heights--the likelihood ratios.

Student: Thank you. I'm not confident I fully understand all of this yet, but you have given me a lot to think about.

Teacher: If you would like to go further, take a look at the Neyman-Pearson Lemma. You are probably ready to understand it now.

---

Synopsis

Many tests that are based on a single statistic like the one in the dialog will call it "$z$" or "$t$". These are ways of hinting what the null histogram looks like, but they are only hints: what we name this number doesn't really matter. The construction summarized by the student, as illustrated here, shows how it is related to the p-value. The p-value is the smallest test size that would cause an observation of $t=0.1$ to lead to a rejection of the null hypothesis.

In this figure, which is zoomed to show detail, the null hypothesis is plotted in solid blue and two typical alternatives are plotted with dashed lines. The region where those alternatives tend to be much larger than the null is shaded in. The shading starts where the relative likelihoods of the alternatives are greatest (at $0$). The shading stops when the observation $t=0.1$ is reached. The p-value is the area of the shaded region under the null histogram: it is the chance, assuming the null is true, of observing an outcome whose likelihood ratios tend to be large regardless of which alternative happens to be true. In particular, this construction depends intimately on the alternative hypothesis. It cannot be carried out without specifying the possible alternatives.

---

For two practical examples of the test described here -- one published, the other hypothetical -- see https://stats.stackexchange.com/a/5408/919. A detailed application of these ideas to testing a median is presented in my post at https://stats.stackexchange.com/a/131489/919.

Answer 3

Before touching this topic, I always make sure that students are happy moving between percentages, decimals, odds and fractions. If they are not completely happy with this then they can get confused very quickly.

I like to explain hypothesis testing for the first time (and therefore p-values and test statistics) through Fisher's classic tea experiment. I have several reasons for this:

(i) I think working through an experiment and defining the terms as we go along makes more sense that just defining all of these terms to begin with. (ii) You don't need to rely explicitly on probability distributions, areas under the curve, etc to get over the key points of hypothesis testing. (iii) It explains this ridiculous notion of "as or more extreme than those observed" in a fairly sensible manner (iv) I find students like to understand the history, origins and back story of what they are studying as it makes it more real than some abstract theories. (v) It doesn't matter what discipline or subject the students come from, they can relate to the example of tea (N.B. Some international students have difficulty with this peculiarly British institution of tea with milk.)

[Note: I originally got this idea from Dennis Lindley's wonderful article "The Analysis of Experimental Data: The Appreciation of Tea & Wine" in which he demonstrates why Bayesian methods are superior to classical methods.]

The back story is that Muriel Bristol visits Fisher one afternoon in the 1920's at Rothamsted Experimental Station for a cup of tea. When Fisher put the milk in last she complained saying that she could also tell whether the milk was poured first (or last) and that she preferred the former. To put this to the test he designed his classic tea experiment where Muriel is presented with a pair of tea cups and she must identify which one had the milk added first. This is repeated with six pairs of tea cups. Her choices are either Right (R) or Wrong (W) and her results are: RRRRRW.

Suppose that Muriel is actually just guessing and has no ability to discriminate whatsoever. This is called the Null Hypothesis. According to Fisher the purpose of the experiment is to discredit this null hypothesis. If Muriel is guessing she will identify the tea cup correctly with probability 0.5 on each turn and as they are independent the observed result has 0.5$^6$ = 0.016 (or 1/64). Fisher then argues that either:

(a) the null hypothesis (Muriel is guessing) is true and an event of small probability has occurred or,

(b) the null hypothesis is false and Muriel has discriminatory powers.

The p-value (or probability value) is the probability of observing this outcome (RRRRRW) given the null hypothesis is true - it's the small probability referred to in (a), above. In this instance it's 0.016. Since events with small probabilities only occur rarely (by definition) situation (b) might be a more preferable explanation of what occurred than situation (a). When we reject the null hypothesis we're in fact accepting the opposite hypothesis which is we call the alternative hypothesis. In this example, Muriel has discriminatory powers is the alternative hypothesis.

An important consideration is what do we class as a "small" probability? What's the cutoff point at which we're willing to say that an event is unlikely? The standard benchmark is 5% (0.05) and this is called the significance level. When the p-value is smaller than the significance level we reject the null hypothesis as being false and accept our alternative hypothesis. It is common parlance to claim a result is "significant" when the p-value is smaller than the significance level i.e. when the probability of what we observed occurring given the null hypothesis is true is smaller than our cutoff point. It is important to be clear that using 5% is completely subjective (as is using the other common significance levels of 1% and 10%).

Fisher realised that this doesn't work; every possible outcome with one wrong pair was equally suggestive of discriminatory powers. The relevant probability for situation (a), above, is therefore 6(0.5)^6 = 0.094 (or 6/64) which now is not significant at a significance level of 5%. To overcome this Fisher argued that if 1 error in 6 is considered evidence of discriminatory powers then so is no errors i.e. outcomes that more strongly indicate discriminatory powers than the one observed should be included when calculating the p-value. This resulted in the following amendment to the reasoning, either:

(a) the null hypothesis (Muriel is guessing) is true and the probability of events as, or more, extreme than that observed is small, or

(b) the null hypothesis is false and Muriel has discriminatory powers.

Back to our tea experiment and we find that the p-value under this set-up is 7(0.5)^6 = 0.109 which still is not significant at the 5% threshold.

I then get students to work with some other examples such as coin tossing to work out whether or not a coin is fair. This drills home the concepts of the null/alternative hypothesis, p-values and significance levels. We then move onto the case of a continuous variable and introduce the notion of a test-statistic. As we have already covered the normal distribution, standard normal distribution and the z-transformation in depth it's merely a matter of bolting together several concepts.

As well as calculating test-statistics, p-values and making a decision (significant/not significant) I get students to work through published papers in a fill in the missing blanks game.

Answer 4

No amount of verbal explanation or calculations really helped me to understand at a gut level what p-values were, but it really snapped into focus for me once I took a course that involved simulation. That gave me the ability to actually see data generated by the null hypothesis and to plot the means/etc. of simulated samples, then look at where my sample's statistic fell on that distribution.

I think the key advantage to this is that it lets students forget about the math and the test statistic distributions for a minute and focus on the concepts at hand. Granted, it required that I learn how to simulate that stuff, which will cause problems for an entirely different set of students. But it worked for me, and I've used simulation countless times to help explain statistics to others with great success (e.g., "This is what your data looks like; this is what a Poisson distribution looks like overlaid. Are you SURE you want to do a Poisson regression?").

This doesn't exactly answer the questions you posed, but for me, at least, it made them trivial.

Answer 5

A nice definition of p-value is "the probability of observing a test statistic at least as large as the one calculated assuming the null hypothesis is true".

The problem with that is that it requires an understanding of "test statistic" and "null hypothesis". But, that's easy to get across. If the null hypothesis is true, usually something like "parameter from population A is equal to parameter from population B", and you calculate statistics to estimate those parameters, what is the probability of seeing a test statistic that says, "they're this different"?

E.g., If the coin is fair, what is the probability I'd see 60 heads out of 100 tosses? That's testing the null hypothesis, "the coin is fair", or "p = .5" where p is the probability of heads.

The test statistic in that case would be the number of heads.

Now, I assume that what you're calling "t-value" is a generic "test statistic", not a value from a "t distribution". They're not the same thing, and the term "t-value" isn't (necessarily) widely used and could be confusing.

What you're calling "t-value" is probably what I'm calling "test statistic". In order to calculate a p-value (remember, it's just a probability) you need a distribution, and a value to plug into that distribution which will return a probability. Once you do that, the probability you return is your p-value. You can see that they are related because under the same distribution, different test-statistics are going to return different p-values. More extreme test-statistics will return lower p-values giving greater indication that the null hypothesis is false.

I've ignored the issue of one-sided and two-sided p-values here.

Answer 6

What the p-value doesn't tell you is how likely it is that the null hypothesis is true. Under the conventional (Fisher) significance testing framework we first compute the likelihood of observing the data assuming the null hypothesis is true, this is the p-value. It seems intuitively reasonable then to assume the null hypothesis is probably false if the data are sufficiently unlikely to be observed under the null hypothesis. This is entirely reasonable. Statisticians tranditionally use a threshold and "reject the null hypothesis at the 95% significance level" if (1 - p) > 0.95; however this is just a convention that has proven reasonable in practice - it doesn't mean that there is less than 5% probability that the null hypothesis is false (and therefore a 95% probability that the alternative hypothesis is true). One reason that we can't say this is that we have not looked at the alternative hypothesis yet.

Imaging a function f() that maps the p-value onto the probability that the alternative hypothesis is true. It would be reasonable to assert that this function is strictly decreasing (such that the more likely the observations under the null hypothesis, the less likely the alternative hypothesis is true), and that it gives values between 0 and 1 (as it gives an estimate of probability). However, that is all that we know about f(), so while there is a relationship between p and the probability that the alternative hypothesis is true, it is uncalibrated. This means we cannot use the p-value to make quantitative statements about the plausibility of the nulll and alternatve hypotheses.

Caveat lector: It isn't really within the frequentist framework to speak of the probability that a hypothesis is true, as it isn't a random variable - it is either true or it isn't. So where I have talked of the probability of the truth of a hypothesis I have implicitly moved to a Bayesian interpretation. It is incorrect to mix Bayesian and frequentist, however there is always a temptation to do so as what we really want is an quantative indication of the relative plausibility/probability of the hypotheses. But this is not what the p-value provides.

Answer 7

Imagine you have a bag containing 900 black marbles and 100 white, i.e. 10% of the marbles are white. Now imagine you take 1 marble out, look at it and record its colour, take out another, record its colour etc.. and do this 100 times. At the end of this process you will have a number for white marbles which, ideally, we would expect to be 10, i.e. 10% of 100, but in actual fact may be 8, or 13 or whatever simply due to randomness. If you repeat this 100 marble withdrawal experiment many, many times and then plot a histogram of the number of white marbles drawn per experiment, you'll find you will have a Bell Curve centred about 10.

This represents your 10% hypothesis: with any bag containing 1000 marbles of which 10% are white, if you randomly take out 100 marbles you will find 10 white marbles in the selection, give or take 4 or so. The p-value is all about this "give or take 4 or so." Let's say by referring to the Bell Curve created earlier you can determine that less than 5% of the time would you get 5 or fewer white marbles and another < 5% of the time accounts for 15 or more white marbles i.e. > 90% of the time your 100 marble selection will contain between 6 to 14 white marbles inclusive.

Now assuming someone plonks down a bag of 1000 marbles with an unknown number of white marbles in it, we have the tools to answer these questions

i) Are there fewer than 100 white marbles?

ii) Are there more than 100 white marbles?

iii) Does the bag contain 100 white marbles?

Simply take out 100 marbles from the bag and count how many of this sample are white.

a) If there are 6 to 14 whites in the sample you cannot reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 6 through 14 will be > 0.05.

b) If there are 5 or fewer whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 5 or fewer will be < 0.05. You would expect the bag to contain < 10% white marbles.

c) If there are 15 or more whites in the sample you can reject the hypothesis that there are 100 white marbles in the bag and the corresponding p-values for 15 or more will be < 0.05. You would expect the bag to contain > 10% white marbles.

In response to Baltimark's comment

Given the example above, there is an approximately:-

4.8% chance of getter 5 white balls or fewer

1.85% chance of 4 or fewer

0.55% chance of 3 or fewer

0.1% chance of 2 or fewer

6.25% chance of 15 or more

3.25% chance of 16 or more

1.5% chance of 17 or more

0.65% chance of 18 or more

0.25% chance of 19 or more

0.1% chance of 20 or more

0.05% chance of 21 or more

These numbers were estimated from an empirical distribution generated by a simple Monte Carlo routine run in R and the resultant quantiles of the sampling distribution.

For the purposes of answering the original question, suppose you draw 5 white balls, there is only an approximate 4.8% chance that if the 1000 marble bag really does contain 10% white balls you would pull out only 5 whites in a sample of 100. This equates to a p value < 0.05. You now have to choose between

i) There really are 10% white balls in the bag and I have just been "unlucky" to draw so few

or

ii) I have drawn so few white balls that there can't really be 10% white balls (reject the hypothesis of 10% white balls)

Answer 8

In statistics you can never say something is absolutely certain, so statisticians use another approach to gauge whether a hypothesis is true or not. They try to reject all the other hypotheses that are not supported by the data.

To do this, statistical tests have a null hypothesis and an alternate hypothesis. The p-value reported from a statistical test is the likelihood of the result given that the null hypothesis was correct. That's why we want small p-values. The smaller they are, the less likely the result would be if the null hypothesis was correct. If the p-value is small enough (ie,it is very unlikely for the result to have occurred if the null hypothesis was correct), then the null hypothesis is rejected.

In this fashion, null hypotheses can be formulated and subsequently rejected. If the null hypothesis is rejected, you accept the alternate hypothesis as the best explanation. Just remember though that the alternate hypothesis is never certain, since the null hypothesis could have, by chance, generated the results.

Answer 9

I am bit diffident to revive the old topic, but I jumped from here, so I post this as a response to the question in the link.

The p-value is a concrete term, there should be no room for misunderstanding. But, it is somehow mystical that colloquial translations of the definition of p-value leads to many different misinterpretations. I think the root of the problem is in the use of the phrases "at least as adverse to null hypothesis" or "at least as extreme as the one in your sample data" etc.

For instance, Wikipedia says

...the p-value is the probability of obtaining the observed sample results (or a more extreme result) when the null hypothesis is actually true.

Meaning of $p$-value is blurred when people first stumble upon "(or a more extreme result)" and start thinking "more extreeeme?".

I think it is better to leave the "more extreme result" to something like indirect speech act. So, my take is

The p-value is the probability of seeing what you see in a "imaginary world" where the null hypothesis is true.

To make the idea concrete, suppose you have sample x consisting of 10 observations and you hypothesize that the population mean is $\mu_0=20$. So, in your hypothesized world, population distribution is $N(20,1)$.

x
#[1] 20.82600 19.30229 18.74753 18.99071 20.14312 16.76647
#[7] 18.94962 17.99331 19.22598 18.68633

You compute t-stat as $t_0=\sqrt{n}\frac{\bar{X}-\mu_0}{s}$, and find out that

sqrt(10) * (mean(x) - 20) / sd(x)  
#-2.974405

So, what is the probability of observing $|t_0|$ as large as 2.97 ( "more extreme" comes here) in the imaginary world? In the imaginary world $t_0\sim t(9)$, thus, the p-value must be $$p-value=Pr(|t_0|\geq 2.97)= 0.01559054$$

2*(1 - pt(2.974405, 9))
#[1] 0.01559054

Since p-value is small, it is very unlikely that the sample x would have been drawn in the hypothesized world. Therefore, we conclude that it is very unlikely that the hypothesized world was in fact the actual world.

Answer 10

I have also found simulations to be a useful in teaching.

Here is a simulation for the arguably most basic case in which we sample $n$ times from $N(\mu,1)$ (hence, $\sigma^2=1$ is known for simplicity) and test $H_0:\mu=\mu_0$ against a left-sided alternative.

Then, the $t$-statistic $\text{tstat}:=\sqrt{n}(\bar{X}-\mu_0)$ is $N(0,1)$ under $H_0$, such that the $p$-value is simply $\Phi(\text{tstat})$ or pnorm(tstat) in R.

In the simulation, it is the fraction of times that data generated under the null $N(\mu_0,1)$ (here, $\mu_0=2$) yields sample means stored in nullMeans that are less (i.e., ``more extreme'' in this left-sided test) than the one calculated from the observed data.

# p value
set.seed(1)
reps <- 1000
n <- 100      
mu <- 1.85 # true value
mu_0 <- 2 # null value
xaxis <- seq(-3, 3, length = 100)

X <- rnorm(n,mu)

nullMeans <- counter <- rep(NA,reps)

yvals <- jitter(rep(0,reps),2)

for (i in 1:reps)
{  
  tstat <- sqrt(n)*(mean(X)-mu_0) # test statistic, N(0,1) under the given assumptions

  par(mfrow=c(1,3))
  plot(xaxis,dnorm(xaxis),ylab="null distribution",xlab="possible test statistics",type="l")
  points(tstat,0,cex=2,col="salmon",pch=21,bg="salmon")

  X_null <- rnorm(n,mu_0) # generate data under H_0
  nullMeans[i] <- mean(X_null)

  plot(nullMeans[1:i],yvals[1:i],col="blue",pch=21,xlab="actual means and those generated under the null",ylab="", yaxt='n',ylim=c(-1,1),xlim=c(1.5,2.5))
  abline(v=mu_0,lty=2)
  points(mean(X),0,cex=4,col="salmon",pch=21,bg="salmon")

  # counts 1 if sample generated under H_0 is more extreme:
  counter[i] <- (nullMeans[i] < mean(X)) # i.e. we test against H_1: mu < mu_0
  barplot(table(counter[1:i])/i,col=c("green","red"),xlab="more extreme mean under the null than the mean actually observed")

  if(i<10) locator(1)
}
mean(counter)
pnorm(tstat)

Answer 11

I find it helpful to follow a sequence in which you explain concepts in the following order: (1) The z score and proportions above and below the z score assuming a normal curve. (2) The notion of a sampling distribution, and the z score for a given sample mean when the population standard deviation is known (and thence the one sample z test) (3) The one-sample t-test and the likelihood of a sample mean when the population standard deviation is unknown (replete with stories about the secret identity of a certain industrial statistician and why Guinness is Good For Statistics). (4) The two-sample t-test and the sampling distribution of mean differences. The ease with which introductory students grasp the t-test has much to do with the groundwork that is laid in preparation for this topic.

/* instructor of terrified students mode off */

Answer 12

I have yet to prove the following argument so it might contain errors, but I really want to throw in my two cents (Hopefully, I'll update it with a rigorous proof soon). Another way of looking at the $p$-value is

$p$-value - A statistic $X$ such that $$\forall 0 \le c \le 1, F_{X|H_0}(\inf\{x: F_{X|H_0}(x) \ge c\}) = c$$ where $F_{X|H_0}$ is the distribution function of $X$ under $H_0$.

Specifically, if $X$ has a continuous distribution and you're not using approximation, then

1. Every $p$-value is a statistic with a uniform distribution on $[0, 1]$, and
2. Every statistic with a uniform distribution on $[0, 1]$ is a $p$-value.

You may consider this a generalized description of the $p$-values.

Answer 13

What does a "p-value" mean in relation to the hypothesis being tested?

In an ontological sense (what is truth?), it means nothing. Any hypothesis testing is based on untested assumptions. This are normally part of the test itself, but are also part of whatever model you are using (e.g. in a regression model). Since we are merely assuming these, we cannot know if the reason why the p-value is below our threshold is because the null is false. It is a non sequitur to deduce unconditionally that because of a low p-value we must reject the null. For instance, something in the model could be wrong.

In an epistemological sense (what can we learn?), it means something. You gain knowledge conditional on the untested premises being true. Since (at least until now) we cannot prove every edifice of reality, all our knowledge will be necessarily conditional. We will never get to the "truth".

Answer 14

I think that examples involving marbles or coins or height-measuring can be fine for practicing the math, but they aren't good for building intuition. College students like to question society, right? How about using a political example?

Say a political candidate ran a campaign promising that some policy will help the economy. She was elected, she got the policy enacted, and 2 years later, the economy is booming. She's up for re-election, and claims that her policy is the reason for everyone's prosperity. Should you re-elect her?

The thoughtful citizen should say "well, it's true that the economy is doing well, but can we really attribute that to your policy?" To truly answer this, we must consider the question "would the economy have done well in the last 2 years without it?" If the answer is yes (e.g. the economy is booming because of some new unrelated technological development) then we reject the politician's explanation of the data.

That is, to examine one hypothesis (policy helped the economy), we must build a model of the world where that hypothesis is null (the policy was never enacted). We then make a prediction under that model. We call the probability of observing this data in that alternate world the p-value. If the p-value is too high, then we aren't convinced by the hypothesis--the policy made no difference. If the p-value is low then we trust the hypothesis--the policy was essential.

Answer 15

The p-value isnt as mysterious as most analysts make it out to be. It is a way of not having to calculate the confidence interval for a t-test but simply determining the confidence level with which null hypothesis can be rejected.

ILLUSTRATION. You run a test. The p-value comes up as 0.1866 for Q-variable, 0.0023 for R-variable. (These are expressed in %).

If you are testing at a 95% confidence level to reject the null hypo;

for Q: 100-18.66= 81.34%

for R: 100-0.23= 99.77%.

At a 95% confidence level, Q gives an 81.34% confidence to reject. This falls below 95% and is unacceptable. ACCEPT NULL.

R gives a 99.77% confidence to reject null. Clearly above the desired 95%. We thus reject the null.

I just illustrated the reading of the p-value through a 'reverse way' of measuring it up to the confidence level at which we reject the null hypo.