Understanding p-values using an example

Question

Definition of p-values: A p-value is a probability that provides a measure of the evidence against the null hypothesis provided by the sample. Smaller p-values indicate more evidence against null hypothesis. Can someone please explain this in simpler terms or in a language easy to understand?

I know there might already be tons of questions around understanding the interpretation of p-values, however I would ask the question in a very limited form and with the use of a specific example:

A manufacturing company fills up can with mean weight of 3 pounds, the level of significance is assumed to be 0.01

H(0) : u >= 3 -- Null hypotheses
H(a) : u < 3  -- Alternate hypotheses

We are trying to perform a one tailed test for the case where the population standard deviation is known, so for a sample mean of 2.92 and a standard error of 0.03, we get the z-score as -2.67, giving us the probability (p-value) of 0.0038 or 0.38% that the sample mean would be equal to or lower than 2.92.

Since the probability of getting a sample mean equal or less than 2.92 is 0.38%, which is very small, doesn't it mean that we should accept the null hypotheses? As the chances of getting a mean of 2.92 from a sample is only 0.38%.

Or am I completely missing something here?

---

Edit - It has been three days now since I tried understanding hypothesis testing and I think I am almost there, I will try to articulate what I have understood so far and then let me know if there are still any gaps in my understanding

p-values measure the likelihood of obtaining the sample mean that we obtained given that the null hypothesis is true. So for the example that I mentioned, the probability of obtaining a sample mean of 2.92 is 0.038 if that population's mean is 3 (as assumed by the null hypothesis).

Now there could be two reasons for obtaining means of 2.92:

1. The assumed population mean (i.e., the null hypothesis) is not correct, or
2. the population mean is 3 but due to a sampling error / an unlikely sample we got a mean of 2.92.

Now if we select statement 1, we run the chance of making type 1 error and this is where the level of significance comes into play. Using the level of significance we can see if we can reject the null hypothesis or cannot reject null hypothesis.

Answer 1

Imagine you could measure the weight of all cans that the manufacturing company has ever made and the mean would be $2.87$ pounds. Then imagine you would take 10 cans randomly and see how much they weight. It is unlikely to get the exact mean of all cans ($2.87$ pounds), hence you end up with a mean of $2.65$, for example. If you would do that again and again - taking 10 cans and measuring the weight - you would get a distribution of means. The best guess about the true mean is the mean of the distribution you obtained. Extreme values like $1.9$ or $3.5$ pounds will be unlikely and even more extreme values will be even more unlikely.

Doing significance tests usually means that you look how likely the mean you observed is if you assume that your sample was drawn from a population with mean zero. If the mean that you observed is very unlikely you would decide to discard the null hypothesis. The only difference between what I have said so far and your example is that you assume the null hypothesis a mean of $\ge 3$. So the $0.38\%$ you report say that the probability of getting your mean of $2.92$ from a population with a mean of $\ge 3$ is so unlikely that you would discard the null hypothesis and accept the alternative hypothesis which is $<3$. Your evidence indicate that the cans weight less than $3$ pounds.

This means it is the opposite: having a $p$ of $0.38\%$ as you report doesn't mean you have to keep the null hypothesis because your result is so unlikely but it means that you can discard the null hypothesis because your data was very unlikely a randomly drawn sample from a population with a mean of $3$ (i.e., your data would be very unlikely given that the null hypothesis is true).

Answer 2

Here is a figure that shows your problem on two scales: at left is the original scale in terms of pounds; at right is the standard of z-scale often used in testing.

To begin, let's look at your problem in terms of the fixed significance level $\alpha = 0.01 = 1\%.$ In the right-hand panel, your $Z$-score is shown at the heavy vertical bar at $-2.67.$ The "critical value" for a test at the 1% level is shown by the vertical dotted line at $-2.326,$ which cuts 1% of the probability from the lower tail of the standard normal distribution.

Because the $Z$-score is to the left of the critical value, one rejects the null hypothesis at level $\alpha = 1\%.$ The P-value is the probability under the standard normal curve to the left of the heavy blue line. That area is smaller than $1\%,$ so in terms of P-values, we reject $H_0$ when the P-value is smaller than $1\%.$

You can see that the left-hand plot is the same as the right-hand plot, except for scale. It is not possible to make a printed normal table for all possible normal distributions. By converting to $Z$-scores we can always use a single printed table for the 'standard' normal distribution, which has mean 0 and standard deviation 1.

If we were going to do this production-monitoring procedure repeatedly with $n = 36$ observations each time, then we could find the critical value on the 'pound' scale; it is at 2.581 pounds. (That's because $(2.581 - 3)/.18 = -2.236,$ where the $0.18$ is the standard error.) Then we could turn the testing job over to a non-statistician, with instructions: "If the average weight for 36 cans is less than 2.581 pounds, let me know because we aren't putting enough stuff in our cans." (Or if we can't even trust the non-statistician with averages, the criterion might be a total weight less than 92.92 pounds.)

Answer 3

Since your question is actually quite precise, I would like to keep it rather concise.

Definition of p-value: the p-value is the probability of the data (or even more extrem data) given the null hypothesis is actually true.

If this probability is high, then there is no reason why we should reject the null hypothesis: the data is perfectly in line with the null hypothesis. If the p-value is small, then the data seems implausible given the null hypothesis. The more implausible the data, the stronger our evidence against the null.

A level of significance of 0.01 means: to reject the null hypothesis, the probability of the data must be less than 1%. If the null hypothesis is actually true, we have therefore 1% chance to see data, which is so implausible that we would wrongly reject the null hypothesis.

Regarding your example: there is only 0.38% chance to see this data, if the null hypothesis is true, which is below our threshold of significance. Hence, the data seems very unlikely, and therefore we conclude that we don't believe in the null hypothesis anymore.

Answer 4

Assume the significance level is $\alpha$, which when talking about the null hypothesis, we are usually looking at 5% or 1% and so on.

In simple terms: p-value is the smallest $\alpha$ at which we reject the null hypothesis.

So, when your p-value is 0.15, then we accept the null hypothesis when $\alpha$ is 5% (or our confidence interval is 90%). But change that to only have a confidence interval of 60% and you reject your null hypothesis. Similarly, when your p-value=0.0038, it means you accept the null hypothesis under any value smaller than < 0.38%. That's why you compare p-value with $\alpha$ and if p-value < $\alpha$, you say that you cannot accept the null hypothesis.