I have gone through the definition of Fisher's method in several materials but it is unclear how Fisher's method is stronger than individual tests. I guess it is not always the case, i.e., there are scenarios that the combined test may perform worse, but not sure in which scenarios. To verify it, I tried setting up the following simple setting and want to compare type-II errors in the individual and combined tests.

Assume there are 2 individual tests having the same hypothesis $H_0: \mu=\mu_0$, $H_a:\mu > \mu_0$ with p-values $p_1$ and $p_2$ respectively. Fisher's method combines the p-values by calculating the following test statistic: $$\chi^2=-2(ln(p_1)+ln(p_2))$$ Let $\mu_a$ be the value that satisfies $H_a$. Let $\beta_f$ and $\beta_1$ respectively be the type-2 error of the combined test and the first test. I want to know in which conditions, $\beta_f<\beta_1$ and vice versa, but am not sure how to continue. Is the above formulation complete or are any further assumptions needed?

  • $\begingroup$ 1. Are you seeking to simulate? Or something else? 2. One thing missing is that you don't seem to have the d.f. on your chi-squared variate; it should be 4 in this case. 3. Ponder this little thought: say you have a sample of size 2m (independent observations) and consider two cases: a: treat it as one large sample. b: you treat it as two samples and combine log-likelihoods by addition (because they're independent, so the combined likelihood is a product). What is the overall likelihood under each scenario? This should give you some clues about what to expect for many tests you might consider $\endgroup$
    – Glen_b
    Commented Jul 11 at 1:56
  • $\begingroup$ I should have mentioned that in the simplest split case you take it as two samples of size m, but perhaps that's a natural enough thing to do. Fisherian testing would typically use the likelihood as the test statistic (or some equivalent), making the analogy to information content from a larger sample or by combining likelihood from two samples particularly pertinent but the conclusions for combining tests work somewhat more generally than when the test is Fisherian in style. $\endgroup$
    – Glen_b
    Commented Jul 11 at 2:05


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.