I hope to work on a hypothesis testing problem to test $\beta$ against the null hypothesis ($H_{null}$) $\beta=0$, and we can consider Wald test with the test statistics $$ W = \dfrac{\beta^2}{var(\beta)} $$ Then, with $W$, I can get p-values, which is essentially "getting a value of the test statistic as extreme as or even more extreme than what is observed by chance alone", so $p(t > W | H_{null})$.

However, the sample size I have to calculate $\beta$ and $var(\beta)$ is fairly small. The results, although p-values for some covariates are small, are not very convincing.

On the other hand, I have some p-values calculated for the same set of covariates, from a larger body of samples. Basically, I have a prior $p_0(t>W_0|H_{null})$. In addition, sometimes I also have $\beta_0$.

I wonder whether/how I can calibrate $p(t > W | H_{null})$ with $p_0(t>W_0|H_{null})$.

Some of my current efforts

  • I learned a way to do this is to simulate a large amount of data that leads to $p_0$ if we conduct the test, and then combine the simulated data and my real ones to do the hypothesis testing, but I hope there are more efficient ways to do it.
  • Bayesian hypothesis testing sounds relevant, but it assumes $p_0(H_{null})$ instead of $p_0(t>W_0|H_{null})$, and it leads to a MAP result. I hope we can still have a p-value at the end.

1 Answer 1


It sounds like what you are after is a meta-analysis. This is the frequentist analog to a Bayesian analysis using an informative prior.

A meta-analysis can come in many forms. One option is to combine the subject-level observations into a single data set and run a single model to produce p-values and confidence intervals. This is the ideal approach. Another option is to combine summary-level statistics or estimators into a single value and estimate its standard error. One can then perform a hypothesis test to produce a p-value and invert a hypothesis test to construct a confidence interval. Another option is in line with what you proposed, which is to use simulated data that mimics the observed results from a historical study and use option one.

Here is a related paper on meta-analysis and transfer learning.

  • $\begingroup$ Thanks. Looks like what I hoped to learn is the second option. I will read the paper and follow up. $\endgroup$
    – user42004
    Dec 20, 2021 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.