Stouffer's Z-score method: what if we sum $z^2$ instead of $z$?

Question

I am performing $N$ independent statistical tests with the same null hypothesis, and would like to combine the results into one $p$-value. It seems that there are two "accepted" methods: Fisher's method and Stouffer's method.

My question is about Stouffer's method. For each separate test I obtain a z-score $z_i$. Under a null hypothesis, each of them is distributed with a standard normal distribution, so the sum $\Sigma z_i$ follows a normal distribution with variance $N$. Therefore Stouffer's method suggests to compute $\Sigma z_i / \sqrt{N}$, which should be normally distributed with unit variance, and then use this as a joint z-score.

This is reasonable, but here is another approach that I came up with and that also sounds reasonable to me. As each of $z_i$ comes from a standard normal distribution, the sum of squares $S=\Sigma z^2_i$ should come from a chi-squared distribution with $N$ degrees of freedom. So one can compute $S$ and convert it to a $p$-value using cumulative chi-squared distribution function with $N$ degrees of freedom ($p=1−X_N(S)$, where $X_N$ is the CDF).

However, nowhere can I find this approach even mentioned. Is it ever used? Does it have a name? What would be advantages/disadvantages compared to Stouffer's method? Or is there a flaw in my reasoning?

Answer 1

One flaw that jumps out is Stouffer's method can detect systematic shifts in the $z_i$, which is what one would usually expect to happen when one alternative is consistently true, whereas the chi-squared method would appear to have less power to do so. A quick simulation shows this to be the case; the chi-squared method is less powerful to detect a one-sided alternative. Here are histograms of the p-values by both methods (red=Stouffer, blue=chi-squared) for $10^5$ independent iterations with $N=10$ and various one-sided standardized effects $\mu$ ranging from none ($\mu=0$) through $0.6$ SD ($\mu=0.6$).

The better procedure will have more area close to zero. For all positive values of $\mu$ shown, that procedure is the Stouffer procedure.

---

R code

This includes Fisher's method (commented out) for comparison.

n <- 10
n.iter <- 10^5
z <- matrix(rnorm(n*n.iter), ncol=n)

sim <- function(mu) {
  stouffer.sim <- apply(z + mu, 1, 
                    function(y) {q <- pnorm(sum(y)/sqrt(length(y))); 2*min(q, 1-q)})
  chisq.sim <- apply(z + mu, 1, 
                    function(y) 1 - pchisq(sum(y^2), length(y)))
  #fisher.sim <- apply(z + mu, 1,
  #                  function(y) {q <- pnorm(y); 
  #                     1 - pchisq(-2 * sum(log(2*pmin(q, 1-q))), 2*length(y))})
  return(list(stouffer=stouffer.sim, chisq=chisq.sim, fisher=fisher.sim))
}

par(mfrow=c(2, 3))
breaks=seq(0, 1, .05)
tmp <- sapply(c(0, .1, .2, .3, .4, .6), 
              function(mu) {
                x <- sim(mu); 
                hist(x[[1]], breaks=breaks, xlab="p", col="#ff606060",
                     main=paste("Mu =", mu)); 
                hist(x[[2]], breaks=breaks, xlab="p", col="#6060ff60", add=TRUE)
                #hist(x[[3]], breaks=breaks, xlab="p", col="#60ff6060", add=TRUE)
                })

Answer 2

One general way to gain insight into test statistics is to derive the (usually implicit) underlying assumptions that would lead that test statistic to be most powerful. For this particular case a student and I have recently done this: http://arxiv.org/abs/1111.1210v2 (a revised version is to appear in Annals of Applied Statistics).

To very briefly summarize (and consistent with the simulation results in another answer) Stouffer's method will be most powerful when the "true" underlying effects are all equal; the sum of Z^2 will be most powerful when the underlying effects are normally distributed about 0. This is a slight simplification that omits details: see section 2.5 in the arxiv preprint linked above for more details.

Answer 3

Slightly o/t: one of the issues with both these approaches is the loss of power due to the degrees of freedom (N for stouffer's; 2N for Fisher's). There have been better meta-analytical approaches developed for this, which you may want to consider (inverse-variance weighted meta-analysis, for example).

If you're looking for evidence of some alternative tests within a group, you may want to look at Donoho and Jin's higher criticism statistic: https://projecteuclid.org/euclid.aos/1085408492

Answer 4

To answer the question and for any further readers: is it ever used?, there is an exhaustive paper by Cousins (2008) on arXiv, which listed and reviewed a couple of alternative approaches. The proposed one does not seem to appear.