Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When doing hypothesis testing, we calculate the distribution of test statistic (for example z) under null hypothesis and then compare the actual z (one calculated from our data) to it.
Why can't we instead calculate the distribution of z statistic from our data (by bootstraping mean, for example) (or calculate 95% CI) and compare it to 0, like it is shown below.
I'd also see an example (or a simulation) where this fails, not just a theory behind it.
Why can't we instead calculate the distribution...
You can and it is actually done in a certain way. Depending on how you compute it exactly, your second distribution can have different interpretations (with a normal distribution the computations coincide).
Bayesian posterior distribution when you estimate the mean parameter $\mu$ of a normal distribution with known variance, and you use an improper uniform distribution as prior, then you get as posterior distribution the normal distribution with $\hat\mu = \bar{x}$ as mean and $\hat{\sigma}^2 = \sigma^2/n$ as variance.
The meaning of this distribution is the probability density of the parameter $\mu$ given the data $x$. It tells where the parameter $\mu$ is likely gonna be.
Frequentist confidence distribution. In your question you suggest to compute the confidence interval and compare this with the 0. This confidence interval is actually indirectly a hypothesis test. The confidence interval is a range of parameters where a hypothesis tests is positive.
From these confidence intervals one could create some distribution by considering multiple confidence intervals of different percentage and see the distribution as the difference between them.
This confidence distribution is not the probability of the position of the parameter $\mu$ given the data $x$*.
But it does have a probabilistic interpretation. It is a probability about the experiments. Conditional on a true parameter, $\mu$, you will cover all quantiles of the confidence interval/distribution with equal probability.
*The difference between the confidence distribution and the posterior distribution is whether you condition on the data or on the true parameter
A fiducial distribution In the comments you suggest that the distribution could be a distribution of the parameter based on a bootstrapping of the data. Thus it will be a distribution of the data based on the data. Such a distribution could be given a fiducial interpretation and the Bootstrapping approximates estimates the probability distribution $P(\bar{X} \leq x)$ for which you could draw, with some handwaving, a probability density function as if it is a distribution of a continuous variable.
In the same way as the confidence distribution (and they are quite the same) this fiducial distribution is not like the posterior distribution a distribution of the parameter given the data.
These distributions coincidence when you assume that the data is normal distributed. But for other distributions this is not the same.
Consider the estimate of a confidence interval for a binomial proportion. If this is done with a normal distribution approximation, plusminus some multiple of the standard deviation/error, then one has a Wald interval but a small mistake is made, the standard deviation for different values of the mean is not the same (and because of this the Wald interval may for instance contain negative values for the proportion). The Wilson score interval corrects for this.
The logic of hypothesis tests asks whether what is observed is plausible under the null model. This requires that probabilities are computed assuming the null model.
If you take the parameter value estimated from the data and then compute the probability for 0 or something further away (which is how I interpret your suggestion), you basically compute whether it would have been plausible to observe an estimated value of 0 given the assumed parameter value that is actually the estimated one. This doesn't make sense to me - why would you want to compute the probability for observing zero (or rather a certain set involving zero) if zero is not what you in fact observed as estimator? (Although your suggestion may be in some cases equivalent to doing something reasonable, see answer by Sextus Empiricus.)
