0
$\begingroup$

I often see the p-value being defined as "A p-value can be defined as the probability of obtaining a test result, or a more extreme one, given that H0 is true". When we define our significance level, alpha (usually 0.05), we're essentially trying to say that there's a 5% risk that we will incorrectly reject it (commiting a type 1 error).

Now, if the goal of many testes (besides normality, etc) is to reject the H0 (p-value < alpha), and given that alpha tells us the risk that we will incorrectly reject H0, how certain are we that when rejecting the H0, we're actually seeing difference in our samples and the samples are not at random?

Let's say population A and B with the same mean of 170. Essentially, H0 would be true, but we could very well take a group of individuals from population A that have a mean of 150 and another group from population B that have a mean of 180 and the resulting test statistic could be high enough so that the p-value would be < than alpha, thus leading to the rejection of H0, although the populations have the same mean (H0: uA = uB)

In other words, if alpha = 0.05, we know that if H0 is true, there’s a 5% risk that we will incorrectly reject it (commiting a type 1 error if H0 is true). So we want the risk of commiting this type 1 error (if H0 is true) to be less than 5% and only then do we accept our test as statistical significant?

I hope I'm making sense and what I said was (partly) correct. I'm just trying to wrap my head around how p-value is actually useful (given that we can very well have a significant value even though it's false - I believe this would be quite bad in clinical trials and I'm assuming the alpha would be even less)

Thank you

Improve this question
$\endgroup$
3
  • $\begingroup$ "to be less than 5%" - typically 'no more than $\alpha$' rather than 'less than'. $\endgroup$
    – Glen_b
    Sep 20, 2022 at 22:21
  • $\begingroup$ If the null hypothesis is always true, then $100\%$ of the time you reject it you are making an error. If the null hypothesis is never true, then $0\%$ of the time you reject it you are making an error. So you have no certainty when rejecting the null hypothesis. You would need to know the prior probability the null hypothesis is true, and if you knew that then you might use Bayesian methods instead of classical frequentist hypothesis testing. $\endgroup$
    – Henry
    Sep 20, 2022 at 22:21
  • $\begingroup$ Thank you for your comments! @Henry, then If alpha can be seen as the risk that we will incorrectly reject H0 when H0 is true (type 1 error), we then want the p-value to be less than alpha so we’re minimizing the risk of rejecting H0 (thus accepting H1) when H0 shouldn’t have been rejected? $\endgroup$
    – Chronicles
    Sep 20, 2022 at 22:31

2 Answers 2

Reset to default
2
$\begingroup$

The given definition of the p-value, "probability of obtaining a test result, or a more extreme one, given that H0 is true", is more or less fine. I'd say "probability of obtaining the observed test result or a more extreme one". This means that if the p-value is very low, we have seen something so extreme that we wouldn't normally expect such a thing under the null hypothesis, and therefore we reject the H0. The level $\alpha$ basically decides how small is too small. $\alpha$ is chosen small and therefore the probability to wrongly reject a true H0 is small.

"...how certain are we that when rejecting the H0, we're actually seeing difference in our samples and the samples are not at random?"

In case your H0 is chosen so that you interpret it as "samples are at random" (which is somewhat ambiguous), the only thing you can be certain about is that something has happened that was not to be expected and will rarely happen under the H0. The only "performance guarantee" is that if you run such tests often, you will only reject a true H0 a proportion of $\alpha$ times.

"but we could very well take a group of individuals from population A that have a mean of 150 and another group from population B that have a mean of 180 and the resulting test statistic could be high enough so that the p-value would be < than alpha, thus leading to the rejection of H0, although the populations have the same mean (H0: uA = uB)"

That's true, however if $\alpha$ small, this will happen very rarely.

" So we want the risk of committing this type 1 error (if H0 is true) to be less than 5% and only then do we accept our test as statistical significant?"

There is some confusion of terminology here. The term "accept" is not normally used for the situation in which the test is significant. The test is not "accepted" but rather defined to be statistically significant in this case, which is just a different way of saying that the H0 is rejected, i.e., incompatible with the data in the sense explained above.

"I'm just trying to wrap my head around how p-value is actually useful (given that we can very well have a significant value even though it's false - I believe this would be quite bad in clinical trials and I'm assuming the alpha would be even less)"

There's no way around this problem though if we have random variation. You may observe means 150 and 180 even if the distribution within the two groups is actually the same. This is just how it is. You can hardly do better than having a decision rule that occasionally but rarely gets things wrong.

(I should say that there are other criticisms of p-values and statistical tests, but I will not comment on them here. Also, as said in a comment, if you want a probability that the H0 is true or not, you need a Bayesian approach, and you need to specify a prior distribution first. The basic idea of a test is very simple and intuitive and hard to replace: If something happens that is very unexpected under H0, this counts as evidence against H0, made more precise by the p-value.)

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you so much for your answer! Really helpful :) So to confirm (as a previous comment I made), "If alpha can be seen as the risk that we will incorrectly reject H0 when H0 is true (type 1 error), we then want the p-value to be less than alpha so we’re minimizing the risk of rejecting H0 (thus accepting H1) when H0 shouldn’t have been rejected? " I'm assuming thats where you say "The level α basically decides how small is too small. α is chosen small and therefore the probability to wrongly reject a true H0 is small" $\endgroup$
    – Chronicles
    Sep 20, 2022 at 22:42
  • 1
    $\begingroup$ @ROO It can be mathematically shown (quite easily) that rejecting the H0 in case that the p-value is $<\alpha$ just defines a decision rule that has a type I error probability of $\alpha$. We're not "minimising the risk", we just keep it as small as specified by the $\alpha$.. $\endgroup$
    – Christian Hennig
    Sep 20, 2022 at 22:47
  • 2
    $\begingroup$ @ROO By the way, I don't like the wording "we then want the p-value to be less than $\alpha$"; as scientists we should be interested in any result, be it significant or not, that tells us about the relation of our data to the null hypothesis. I don't think it leads to anything good to "want" one result rather than another (people tend to make an effort to find what they want, which means it cannot be trusted anymore). $\endgroup$
    – Christian Hennig
    Sep 20, 2022 at 22:50
0
$\begingroup$

In other words, if alpha = 0.05, we know that if H0 is true, there’s a 5% risk that we will incorrectly reject it (commiting a type 1 error if H0 is true). So we want the risk of commiting this type 1 error (if H0 is true) to be less than 5% and only then do we accept our test as statistical significant?

Correct (aside from some infelicity of phrasing).

We have to arrive at some decision rule to use to decide when a test statistic is 'too discrepant' to continue to accept that $H_0$ is a reasonable description of the data.

If a suitable test statistic is chosen, it will tend to behave differently when $H_1$ is true from how it behaves when $H_0$ is the case. We can then choose to reject $H_0$ in those situations that most clearly suggest $H_1$ over $H_0$.

We then are left to choose the boundary between the two alternative decisions (usually this critical value is the least discrepant case for which we'd still reject) which tells us the rejection region for the test statistic. This is usually done by choosing an upper limit on the rate at which we would wrongly reject $H_0$, $\alpha$.

Given the usual approach is to fix $\alpha$, we then would like to choose other aspects of the test to give a good chance to reject $H_0$ when its false (choice of test statistic, choice of rejection region, sample size)

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your answer, @Glen_b! When you mention "This is usually done by choosing an upper limit on the rate at which we would wrongly reject H0, α.", you mean the upper-lower bounds on the tails of a curve (t-distribution, for example), right? In this case, however, if our observed test statistic is greater/lower than our expected test statistic (thus falling in one of the upper/lower regions of the tail), we would reject H0, right? Since it falls on that region so we reject H0 (area under the curve aka p-value < alpha, essentially) Or were you mentioning something else? $\endgroup$
    – Chronicles
    Sep 20, 2022 at 22:53
  • $\begingroup$ The critical value is the cut off between reject/not reject for some test statistic $T$ (e.g. "reject when $|T|\geq t_c$"), though it doesn't have to be that the critical region is in the tails of the distribution of $T$ necessarily. Typically we will choose a statistic so the cases most indicative of $H_1$ will tend to be in the upper or lower tail of the distribution of the statistic or both, but it could happen that there's some particular interval or intervals of the values $T$ could take that is not in the tail but suggest $H_1$, perhaps. In choosing where to place bounds ...ctd $\endgroup$
    – Glen_b
    Sep 20, 2022 at 23:31
  • $\begingroup$ ctd . . . we typically base that on putting a limit on $\alpha$, the rate at which we reject $H_0$ when it's true. Note that with compound nulls (rather than simple, equality nulls), different possible parameter values under $H_0$ may have different type I error rates; we are choosing a bound for the worst case. . $\endgroup$
    – Glen_b
    Sep 20, 2022 at 23:33
  • $\begingroup$ Imagine we were considering a sequence of possible rejection rules. We'd like to organize it so that if we had a less stringent rule (i.e. reject more often), that the rejection region would include all the cases we'd reject for in a more stringent rule (that is, we generally want our rejection regions to be nested within less strict rejection regions, for consistency). ... Also, take care with the use of the word 'expected' in relation to the test statistic. Note also that not all statistics are continuous, so you don't necessarily have a 'curve' - a density - for the test statistic. ... ctd $\endgroup$
    – Glen_b
    Sep 20, 2022 at 23:42
  • $\begingroup$ ctd ... These details aside, you seem to have the basic idea there, yes. $\endgroup$
    – Glen_b
    Sep 20, 2022 at 23:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.