6
$\begingroup$

Wikipedia defines a p-value as follows:

the p-value is the probability of obtaining test results at least as extreme as the result actually observed, under the assumption that the null hypothesis is correct.

And to compute a p-value, we need a sampling distribution.

My question is: when we talk about the null hypothesis here, are we referring to the sampling distribution constructed:

  1. Before any data is collected (a priori), or
  2. After data has been collected (a posteriori)?

Bonus points for providing any reference which speaks to this explicitly.

I think the correct answer is the a priori version, otherwise our type 1 and 2 error rates may deviate from the rates specified during the power analysis. However, I believe most A/B test software calculates the p-value from the sampling distribution generated directly from the sample data collected - so I'm confused.

Quick example

Suppose that I am planning to test a new website landing page, with the aim of improving conversion rates. Applying the Null Hypothesis Significance Testing framework (NHST), I lay out my hypotheses a priori:

  • Null hypothesis (H0): treatment and control samples are drawn from populations with identical proportions: 6% conversion rates
  • Alternate hypothesis (HA): treatment and control samples are drawn from populations with different proportions: 7% and 5% conversion rates, respectively

To achieve a significance level of 0.05 and a power of 0.8, I calculate a minimum required sample size of 2213:

# R code
prop_baseline <- 0.05          
prop_treatment <- 0.07
alpha <- 0.05
beta <- 0.2 

n <- ceiling(
        power.prop.test(
            p1 = prop_baseline, 
            p2 = prop_treatment, 
            sig.level = alpha, 
            power = 1-beta, 
            alternative = c("two.sided")
        )$n
      )
n

# [1]  2213

I also determine the a priori critical value (= 1.4%):

prop_pooled <- (prop_baseline+prop_treatment)/2
se_h0 <- sqrt(prop_pooled * (1-prop_pooled) * (1/n + 1/n))
se_ha <- sqrt((prop_baseline*(1-prop_baseline))/n + 
    (prop_treatment*(1-prop_treatment))/n)
critical_value <- qnorm(alpha / 2, mean = 0, sd = se_h0, 
    lower.tail = FALSE)
critical_value

# [1]  0.0139930354339993

We can visualize the sampling distributions for both hypotheses like so: enter image description here

At the conclusion of the test, the results are: enter image description here

The a priori decision framework:

The effect size (= 0.0154) is greater than the critical value (= 0.0140).

And so, the p-value < 0.05:

2*(1 - pnorm(0.01536376, mean = 0, sd = se_h0))

# [1]  0.0314007090609361

So, we reject the null hypothesis in favor of the alternate hypothesis.

The a posteriori decision framework:

The p-value > 0.05:

prop.test(x=c(141, 175), n=c(n,n))$p.value

# [1]  0.0540496200600176

So, we DO NOT reject the null hypothesis.

What's the correct approach?

$\endgroup$
8
  • 8
    $\begingroup$ This question appears to mix up terms from Bayesian and classical analysis. In the latter -- where p-values are concerned -- "a priori" and "a posteriori" are not ordinarily used. In particular, the null hypothesis is a set of hypothetical distributions. Usually, these distributions aren't even used to describe the data or to deduce anything about the data distribution: they are posited as part of an argument intended to refute the null hypothesis. $\endgroup$
    – whuber
    Nov 27, 2023 at 16:23
  • 2
    $\begingroup$ I agree with @Christian's point in an answer below. I also agree with others who point out that the terms a priori and a posteriori are misleading here, and don't help drill down to the cause of the discrepancy. Finally, both your methods use approximations to convert chunky binomial probabilities to smooth Gaussian distribution. I suspect (but am not sure) that your two methods are using different approximations. $\endgroup$ Nov 27, 2023 at 17:16
  • $\begingroup$ I know that the terms "prior" and "posterior" are synonymous with Bayesian inference. But I have used the phrases "a priori" and "a posteriori" because I have seen them used (specifically in the context of null hypothesis significance testing) in the literature. Still... $\endgroup$
    – Rez99
    Nov 27, 2023 at 21:38
  • $\begingroup$ ...I hear that they're causing confusion. So what language should we use to refer to the analytical steps (within the NHST framework) conducted before data is collected vs after data is collected? $\endgroup$
    – Rez99
    Nov 27, 2023 at 21:38
  • 1
    $\begingroup$ If that's an unfamiliar concept, then please consult an introduction to null hypothesis testing. There are some here on CV. For instance, at stats.stackexchange.com/a/436264/919 I give an account, with references. $\endgroup$
    – whuber
    Nov 27, 2023 at 22:21

5 Answers 5

12
$\begingroup$

The null hypothesis is fixed before looking at the sample (or even planning the sample size). The p-value is obviously computed from the data, but it is based on the test statistic, which normally compares the data with what is expected under the null hypothesis. The null hypothesis is rejected if the "distance" is too big.

Note however that in your example prop.test will test the null hypothesis that the two probabilities are the same and not necessarily 0.06. This is the reason that it doesn't rely on the critical value that you have computed for probability 0.06. So the difference between your two setups is not that one is "a priori" and the other one is "a posteriori" (these terms are not used as a standard in frequentist analyses such as hypthesis tests), but rather that your first test tests the null hypothesis that the probabilities are both equal to 0.06 (so that you can fix the standard error), whereas the second one tests the null hypothesis that both probabilities are equal and can be anything (so that the standard error needs to be estimated from the data; this is apparently what you call "a posteriori").

Both of your test results can make sense; as both relative frequencies (141/2213 and 175/2213) are larger than 0.06, it makes sense to reject the H0 that they are both 0.06, but they may still both be 0.07, say.

In order to fix the sample size in advance, you can choose some specific parameter values in order to do the calculations (you may want to reach a specific power for specific relevant choices of parameters), but given the sample sizes and the data, most tests will test more general null and alternative hypotheses against each other.

$\endgroup$
5
  • 1
    $\begingroup$ +1, its' worth a comment there's a whole separate literature about Bayes hypothesis testing where there's nary a mention of $p$-value. $\endgroup$
    – AdamO
    Nov 27, 2023 at 17:41
  • $\begingroup$ @Christian thanks very much for taking the time to provide this answer. Here's my confusion...you say "The null hypothesis is fixed before looking at the sample". When I read "null hypothesis" I'm visualizing a sampling distribution with a mean and a standard error. We set these parameters ahead of data collection, agreed. But when we go to compute the p-value, we're evaluating our result against a sampling distribution with the same mean (0) but a different standard error (calculated from the sample data). $\endgroup$
    – Rez99
    Nov 28, 2023 at 0:06
  • $\begingroup$ So to me, this is a different sampling distribution and hence a different null hypothesis. Therefore, I don't undersand how we are describing it as "fixed". In fact, it looks like we are updating our hypothesis to match the data, rather than leaving it completely alone (what I called the "a priori" decision framework"). This is where I'm getting stuck :( As a side note I posted an answer in which I explicitly compute the two different standard errors. $\endgroup$
    – Rez99
    Nov 28, 2023 at 0:07
  • $\begingroup$ @Rez99 You seem to be confused about terminology or at least you use it in nonstandard ways. The null hypothesis regards a distribution or a set of distributions for the data you observe. The terms "sampling distribution" and "standard error" apply to a test statistic that is used to test the null hypothesis, but there may be more than one for the same H0, so although the sampling distribution of the test statistic is derived from the H0, these are not the same thing. $\endgroup$ Nov 28, 2023 at 11:23
  • $\begingroup$ @Rez99 There can be test statistics that involve an estimated standard error and some that don't. In the latter case for your situation this comes from the fact that the H0 involves a fixed value 0.6 for the probability so that you can formally write down the standard error without estimating it from the data, whereas if the H0 is "both groups equal" without specifying the probability, the standard error needs to be estimated from the data. This however is not part of the H0 itself, rather of the statistic that is used to test it. $\endgroup$ Nov 28, 2023 at 11:27
6
$\begingroup$

My question is: when we talk about the null hypothesis here, are we referring to the sampling distribution constructed:

Before any data is collected (a priori), or

After data has been collected (a posteriori)?

In a one-sample t-test, for example, it is a combination.

A priori, you decide your null hypothesis, say $\mu=0$ (in general, $\mu = \mu_0$ for some constant $\mu_0$).

A posteriori, you calculate the sample variance, the sample mean, and the sample size.

All four components go into calculating the sampling distribution, test statistic, and p-value.

$$ \dfrac{\bar X - \mu_0}{s/\sqrt{n}} \sim t_{n - 1} $$

These terms a priori and a posteriori allude to and sound like, but are not the same as, the ideas of prior and posterior distributions in Bayesian statistics.

$\endgroup$
4
  • $\begingroup$ Kruschke has some interesting thoughts when it comes to sample size, however. $\endgroup$
    – Dave
    Nov 27, 2023 at 0:48
  • 2
    $\begingroup$ A null hypothesis, from which a p-value is computed, is a very specific subset of the parameter space - with no probability at all assigned to the actual value of the parameter. I think there's a desire to "play" into the language of the question when, really, we should clarify why (as per @whuber's comment) the premise is all wrong. $\endgroup$
    – AdamO
    Nov 27, 2023 at 17:25
  • $\begingroup$ I think it's important to note that the p-value depends on information decided before data collection (the null hypothesis) and information derived from the data (mean, variance, sample size). A p-vale requires all of these pieces of information. After all, the p-value for $H_0: \mu = 0$ is not going to be the same as the p-value for $H_0: \mu = 1$. $\endgroup$
    – Dave
    Nov 27, 2023 at 20:19
  • 1
    $\begingroup$ The Latin phrases a priori and a posteriori do not allude to any concepts in Bayesian inference, though share etymology with prior and posterior. $\endgroup$
    – innisfree
    Nov 28, 2023 at 11:34
3
$\begingroup$

The p-value is computed with the sampling distribution

Given a parameter $\theta \in \Theta$, an observed data vector $\mathbf{x}$ and a test statistic $T$ that is increasing with respect to evidence for the alternative hypothesis (i.e., a higher value is more conducive to the alternative), the p-value for the null hypothesis $H_0: \theta \in \Theta_0$ is defined as:

$$\begin{align} p(\mathbf{x}) &\equiv \sup_{\theta \in \Theta_0} \mathbb{P}(T(\mathbf{X}) \geqslant T(\mathbf{x}) | \theta) \\[12pt] &= \sup_{\theta \in \Theta_0} \ \int \limits_{\mathbf{r} : T(\mathbf{x}) \geqslant T(\mathbf{r})}^\infty f_\mathbf{X}(\mathbf{r}|\theta) \ d \mathbf{r}, \\[6pt] \end{align}$$

As you might be able to see, the p-value is a function of the sampling distribution $f_\mathbf{X}(\ \cdot \ |\theta)$. The sampling distribution is the distribution of the data vector conditional on the parameter, so it is neither the prior distribution (unconditional distribution of the parameter) nor the posterior distribution (conditional distribution of the parameter given the data) used in Bayesian statistics.

$\endgroup$
1
$\begingroup$
  • The $p$-value is a posteriori because it depends on the observed data (or more precisely the observed test-statistic, $T^\star$), being $$ p = \text{Pr}(T \ge T^\star | H_0) $$

  • The type-1 and type-2 error rates are a priori as the type-1 error rate is chosen by the research before collecting data and the type-2 error rate depends only on the experimental design (and the chosen type-1 error rate) and not on the observed data.

  • The outcome of a null hypothesis significance test is a posteriori as the decision on whether or not to reject (at a priori chosen threshold) depends on the observed data.

$\endgroup$
7
  • 1
    $\begingroup$ Seems to me that you are forcing the a priori and posteriori labels onto a system for which those labels don't fit. For example, the p-value depends on the data, yes, but also on the selected statistical model and that model is just as arbitrarily chosen as the type I error rate for a hypothesis test. $\endgroup$ Nov 27, 2023 at 20:35
  • 1
    $\begingroup$ I find the labels 'local' and 'global' to fit better with frequentist significance testing and hypothesis testing. See here for a full explanation: link.springer.com/chapter/10.1007/164_2019_286 $\endgroup$ Nov 27, 2023 at 20:36
  • $\begingroup$ In as much as they mean before and after data (which is the OP’s usage) they fit very well. $\endgroup$
    – innisfree
    Nov 28, 2023 at 11:36
  • $\begingroup$ And FWIW, no researcher I’ve ever met was picking their model or desired error rate arbitrarily $\endgroup$
    – innisfree
    Nov 28, 2023 at 11:39
  • $\begingroup$ There's an arbitrarily chosen model in the original question! See figure 8 in the chapter linked in my previous comment to see how it is easy (and correct, in my opinion) to argue that almost all statistical models are arbitrarily chosen. $\endgroup$ Nov 28, 2023 at 20:31
1
$\begingroup$

I have caused confusion by using the terms "a priori" and "a posteriori". To acknowledge this I will use the phrases "before data collection" and "after data collection" here, instead.

In my question, I identified two separate methods for calculating a p-value (differing by standard error) and I asked which was the correct method to use. For the same set of data, each method produced a different answer and this was of concern to me.

I now believe that the two frameworks provide near identical results (validated via simulation) - it's just that in the example I provided, I had mistakenly included Yates' continuity correction in prop.test(). Once I remove this, the results are very similar:

P-value determined via the null hypothesis sampling distribution (constructed BEFORE data is collected)

prop_baseline <- 0.05          
prop_treatment <- 0.07
n <- 2213
observed_delta <- 0.01536376

print('Standard error of null hypothesis sampling distribution')
se_ha <- sqrt((prop_baseline*(1-prop_baseline))/n + (prop_treatment*(1-prop_treatment))/n)
se_ha

print('P-value')
2 * (1 - pnorm(observed_delta, mean = 0, sd = se_ha))

[1] "Standard error of null hypothesis sampling distribution"
0.00713310288729407
[1] "P-value"
0.0312505476484228

P-value determined via the null hypothesis sampling distribution (constructed AFTER data is collected)

conversions_baseline <- 141
conversions_treatment <- 175
n <- 2213
prop_pooled  <- (conversions_baseline/n + conversions_treatment/n)/2
observed_delta <- 0.01536376


print('Standard error of null hypothesis sampling distribution')
se_ha <- sqrt(prop_pooled * (1-prop_pooled) * (1/n + 1/n))
se_ha
print('P-value manual computation')
2 * (1 - pnorm(observed_delta <- 0.01536376
, mean = 0, sd = se_ha))
print('P-value prop.test computation')
prop.test(x=c(conversions_baseline, conversions_treatment), n=c(n,n), correct = FALSE)$p.value

[1] "Standard error of null hypothesis sampling distribution"
0.00774064826424033
[1] "P-value manual computation"
0.0471649830099237
[1] "P-value prop.test computation"
0.0471649887275255

Now (in contrast to the original example), both methods yield a p-value < 0.05, so we reach the same decision (to reject the null hypothesis) for this particular observation.

Globally, both methods yield very similar power estimates

set.seed(123)

#####################
# Initialize inputs
#####################
prop_baseline <- 0.05          
prop_treatment <- 0.07
alpha <- 0.05
beta <- 0.2 

n <- ceiling(
        power.prop.test(
            p1 = prop_baseline, 
            p2 = prop_treatment, 
            sig.level = alpha, 
            power = 1-beta, 
            alternative = c("two.sided")
        )$n
      )


#####################
# Generate sampling distribution under HA
#####################
num_experiments <- 10000

prop_pooled <- (prop_baseline+prop_treatment)/2
se_h0 <- sqrt(prop_pooled * (1-prop_pooled) * (1/n + 1/n))
se_ha <- sqrt((prop_baseline*(1-prop_baseline))/n + (prop_treatment*(1-prop_treatment))/n)
critical_value <- qnorm(alpha / 2, mean = 0, sd = se_h0, lower.tail = FALSE)

conversions_baseline <- replicate(num_experiments, rbinom(1, size = n, prob = prop_baseline))
conversions_treatment <- replicate(num_experiments, rbinom(1, size = n, prob = prop_treatment))
prop_delta <- conversions_treatment/n - conversions_baseline/n

#####################
# Generate p-values
#####################
p_values <- vector("numeric", length = num_experiments)
for(i in 1:num_experiments){
  p_values[i] <- prop.test(x=c(conversions_baseline[i], conversions_treatment[i]), n=c(n,n), correct = FALSE)$p.value
}

#####################
# Determine power
#####################
print('power_via_sampling_dist_before_data_collection')
power_via_sampling_dist_before_data_collection <- sum(prop_delta>critical_value)/num_experiments
power_via_sampling_dist_before_data_collection
print('power_via_sampling_dist_after_data_collection')
power_via_sampling_dist_after_data_collection <- sum(p_values<0.05)/num_experiments
power_via_sampling_dist_after_data_collection

[1] "power_via_sampling_dist_before_data_collection"
0.8109
[1] "power_via_sampling_dist_after_data_collection"
0.8068

Finally, any discrepancy almost completely disappears when we move away from "chunky binomial probabilities to smooth Gaussian distribution" as per Harvey Motulsky's comment.

$\endgroup$
7
  • $\begingroup$ I'm glad that my suggestion in a comment solved one of the issues (using equivalent approximations in the two methods). But I don't think that solves the problem entirely. In your first method, you ASSUME proportions for the two groups and compute a p-value from that. In the second method, you calculate the p-value for the actual data. That is the only method that makes sense. Why would you want to know the p-value for made-up (assumed) data? You need that framework to compute sample size (lots of "what ifs" in power and sample size calculation. But actual data analysis is based on...data. $\endgroup$ Nov 28, 2023 at 2:06
  • $\begingroup$ I'm not sure it changes that much about the general understanding of the problem if for your specific data set results are not quite as different as they seemed before. It's still two different things and in general they will lead to different results. And still the difference is that one approach fixes the probability to 0.06 as part of the H0, whereas the other H0 doesn't specify it. (You probably can still find examples with more diverse results if you look at data where both empirical probabilities are similar but farther away from 0.06.) $\endgroup$ Nov 28, 2023 at 11:33
  • $\begingroup$ These discussions are helping to move my understanding forward (albeit slowly, sorry), so thank-you both for your patience. I’m not quite there yet, but I can at least see a (perhaps helpful) distinction... $\endgroup$
    – Rez99
    Nov 28, 2023 at 20:32
  • $\begingroup$ In your views of the problem (please keep me honest), the sampling distribution (of the difference in proportions, given the null hypothesis) is a standard normal distribution (SND). There is only 1 SND, of course, so it does not change before/after data collection. With this lens, I can see how me referencing two distinct sampling distributions is confusing. Moreover, the standard error (SE) derived from the sample data is captured entirely within the test statistic (as the denominator), so it does not impact the sampling distribution. $\endgroup$
    – Rez99
    Nov 28, 2023 at 20:32
  • $\begingroup$ In contrast, I'm looking through a non-standardized lens (I find dividing everything by the SE to be yet another layer of computation / abstraction away from the real world). Through this lens, I can physically see 2 distinct sampling distributions (one postulated before data collection, the other formed from the data itself), each with a different width (SE). $\endgroup$
    – Rez99
    Nov 28, 2023 at 20:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.