If I can calculate the probability of null hypothesis $H_0$ being true, based on posterior observations x,

that is, if I can estimate $P(H_0 | x)$,

is there a way to calculate standard error(se) or standard deviation(sd) of this estimate($P(H_0 | x)$)?

which values are needed to calculate se or sd?


A couple of comments: first, you aren't using "posterior observations", but just "observations". Second, as you state in your first line, you aren't estimating $P(H_0|x)$ but calculating it. There isn't any standard error or standard deviation associated with it, as it's a calculation conditional upon the observed data. Given that it's conditional upon the observed data, there's no randomness anywhere to form the basis of a standard deviation of $P(\cdot)$.

Now if you wanted to see how $P(H_0|x)$ might vary across other (hypothetical, unobserved) samples $x_i$, there are a few approaches you can take. One would be to bootstrap the calculation by randomly sampling from $x$ appropriately, say, a thousand or so times, and seeing how $P(H_0|x_i)$ varies. Without knowledge of the details of your problem, it's impossible to go into any depth about how to do this in an appropriate manner. One way of bootstrapping (case resampling) is to randomly sample the rows of $x$ with replacement many times, generating pseudo-samples $x_i$, then recalculate $P(H_0|x_i)$ and use those numbers as the basis for subsequent analysis.

Another bootstrap method (parametric bootstrap) is to generate new random data $y_i$ from appropriately combining the data distributions $P(y|\theta_0,H_0)$ and $P(y|\theta_a,H_a)$, the posterior distributions of the parameters $P(\theta_0|x, H_0)$, $P(\theta_a|x, H_a)$, and the posterior distribution of the model/hypotheses $P(H_0|x)$. (Here I'm assuming there's only one alternative, but if there are more, this approach generalizes in the obvious way.) You would then take these randomly-generated data and re-estimate $P(H_0|y_i)$ many times, collecting the estimates and analyzing them as you wish.

A straightforward way of doing the latter is to do it sequentially:

  1. Select which model / hypothesis is true for data set $i$ by sampling from $P(H_0|x)$,
  2. Randomly generate $\theta_0$ or $\theta_a$, whichever is appropriate given the results of step 1,
  3. Randomly generate the data $y_i$ from $P(y|\theta_0,H_0)$ or $P(y|\theta_a,H_a)$, whichever is appropriate,
  4. Perform all the calculations necessary to generate $P(H_0|y_i)$.

... and iterate over $i$ many times.

The first method of bootstrapping is probably simpler, but if you don't have much data (how much is enough is problem-dependent) you may be better off going the second route. There are other options, though, and you should definitely research the topic a little (the wikipedia page linked to above gives a brief overview) before making a choice.

Note, however, that the results you get will not be mathematically formal results in the same way "the standard error of the sample mean = $\hat{\sigma}/\sqrt{n}$" is a formal result, but estimates based on randomly-generated numbers whose representativeness of the underlying population depends entirely on how representative the original sample $x$ is as well as upon your model specification being at least somewhat accurate.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.