4
$\begingroup$

Kruschke (2014) shows in his book how to compute posterior distributions of effect sizes (standardized mean difference) for the Bayesian analogues of frequentist independent-samples t-tests, and one-way ANOVAs, but there is no mention of effect sizes for more complex designs. One could always run the Bayesian equivalent of a t-test after the ANOVA to get the effect size, but then the effect size estimate would not benefit from "shrinkage" effects of hierarchical models, so it would be best to get estimates of the effect sizes directly from the hierarchical model.

The Bayesian hierarchical model provides estimates of mean differences. The issue I'm having is in the choice of the standardizer to compute standardized mean differences (SMDs). Olejnik and Algina (2000) and Gillett (2003) provide guidelines for computing SMDs that are metrically comparable (i.e. estimate the same magnitudes no matter what the design of the study is) from factorial designs in frequentist settings. For example, suppose that I run a 2x2 design with factors A and B. I want to compute the SMD between two levels of factor A. Factor B is an experimentally "manipulated" factor, then I can use the square root of the within-cell variance (the mean squared error, MSE) as the standardizer. If B is a "stratified" factor (e.g. males vs females), then I need to "add back" the variance explained by the "stratified" factor to the residuals error variance in order to get the appropriate standardizer. In a repeated-measures design, I would also need to "add" back the variance explained by subjects to the residual error variance in order to get a metrically comparable effect size. My question is, how do I get these standardizers from the Bayesian hierarchical model?

Let's take for example the split-plot model of Kruschke:

for (r in 1:Ntotal) {
  y[r] ~ dnorm(mu[r], 1/sigma^2)
  mu[r] <- (a0 + aB[xBetween[r]] + aW[xWithin[r]] + aS[xSubject[r]] 
             + aBxW[xBetween[r], xWithin[r]])
}
sigma ~ dunif(ySD/100, ySD*10)
a0 ~ dnorm(yMean, 1/(ySD*5)^2)
for (i in 1:NxBetweenLvl) { aB[i] ~ dnorm(0.0, 1/sigmaB^2) }
sigmaB ~ dgamma(agammaShRa[1],agammaShRa[2])
for (j in 1:NxWithinLvl) { aW[j] ~ dnorm(0.0, 1/sigmaW^2) }
sigmaW ~ dgamma(agammaShRa[1],agammaShRa[2])
for (k in 1:NxSubjectLvl) { aS[k] ~ dnorm(0.0, 1/sigmaS^2) }
sigmaS ~ dgamma(agammaShRa[1],agammaShRa[2])
for (i in 1:NxBetweenLvl) { for (j in 1:NxWithinLvl) {
  aBxW[i,j] ~ dnorm(0.0, 1/sigmaBxW^2)
} }
sigmaBxW ~ dgamma(agammaShRa[1],agammaShRa[2])

suppose that I'm interested in computing the SMD between two levels of the within-subject factor, and the between-subject factor is a "stratified" factor. To get the standardizer I could take the square root of sigma^2+sigmaS^2+sigmaB^2+sigmaBxW^2. However, while this seems appropriate to add back the variance due to subjects (which is a random effect), I doubt it is appropriate to add back the variance due to the between-subject factor, because "sigmaB" estimates the superpopulation variance (Gelman, 2005). Would it be appropriate to use instead the finite population variance (that is var(aB[]) or var(bB[]) using the coefficients constrained to sum to zero) described by Gelman et al. (2005) in this case?

The situation may be more complex for SMDs of interaction effects in which I'm also often interested. But my question is already quite long, and I would like to understand clearly the simpler cases before moving on to more complex ones.

References

Gelman, A., Tjur, T., McCullagh, P., Hox, J., Hoijtink, H., & Zaslavsky, A. M. (2005). Analysis of variance - Why it is more important than ever. Annals of Statistics, 33(1), 1–53.

Gillett, R. (2003). The metric comparability of meta-analytic effect-size estimators from factorial designs. Psychological Methods, 8(4), 419–433. http://doi.org/10.1037/1082-989X.8.4.419

Kruschke, J. (2014). Doing Bayesian Data Analysis: A Tutorial with R, JAGS, and Stan.

Olejnik, S., & Algina, J. (2000). Measures of Effect Size for Comparative Studies: Applications, Interpretations, and Limitations. Contemporary Educational Psychology, 25, 241–286. http://doi.org/10.1006/ceps.2000.1040

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.