Based on the same random sample and same test statistic, will the p value remain the same even when the distribution of test statistic under null hypothesis is changed. Or is this scenario true only in case of a distribution and it's asymptotic distribution say t and normal?

  • 3
    $\begingroup$ What do you mean by "distribution of null hypothesis is changed"? Are you asking if testing different hypothesis could lead to different p-values? The answer is pretty obvious... $\endgroup$ – Tim Oct 31 '19 at 18:29
  • $\begingroup$ I meant distribution of test statistic under null hypothesis $\endgroup$ – Harry Nov 1 '19 at 3:41

The p-value depends on the sampling distribution of the statistic under the null hypothesis. If one null hypothesis implies a different distribution (i.e., with a different mean or variance) from another null hypothesis, then the p-values will change. If the standard error of the statistic depends on the distribution under the null hypothesis, then the p-value, which depends on the standard error, will differ.


Your question has a lot to do with the subtle nature of what a p-value actually is and how it is interpreted. It's not exactly correct to say that the null hypothesis has a distribution at all; as @Noah correctly points out, the p-value relies on the sampling distribution of the test statistic: in frequentist inference, the sampling distribution is constructed partly from a hypothetical, user-supplied "null hypothesis" (such as 'the true mean $\mu=2$' in a one-sample test or '$\mu_1 - \mu_2 = 0$' in a two-sample test) that involves a statement concerning one or more of the parameters of a probability distribution. Other parameters needed to fully specify the distribution (such as the variance $\sigma^2$ in a model where normality is assumed) might also be hypothesized by the investigator, or they may be estimated using the observed data. Most likely, a technique such as maximum likelihood estimation is used to fix the distributional parameters at their "most likely" values.

What this means in regards to your question is that the p-value relies on a distribution which is partly assumed by the researcher, partly determined by the nature of the researcher's hypothesis, and partly estimated from the data. The p-value may not stay the same if either the researcher's basic assumptions or hypotheses change.

In general, however, formulating a null hypothesis is not an arbitrary process and many statistical tests are robust to mild deviations from normality (which is a common assumption). It is hard to imagine a scenario where a well-formed hypothesis changes within an experiment or a researcher makes an assumption such as normality without first having some evidence that this is a safe assumption to make.

With this in mind, I generally think of frequentist inference as being fully determined by the observed data, while knowing that this isn't exactly true. However, as long as a researcher takes the time to understand their hypotheses and assumptions, there should be no need to change them. IMHO it is quite a slippery slope to think of p-values as things that can be manipulated with a cleverly-formulated null hypothesis or a subtle assumption.

  • $\begingroup$ Excellent answer. $\endgroup$ – Noah Oct 31 '19 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.