3
$\begingroup$

I am conducting a meta-analysis on the effectiveness of a learning intervention. In the studies I analyse, there are multiple outcome measures: the majority of them have a positive direction, i.e. we expect them to increase (e.g., accuracy, proficiency, vocabulary-size, etc.), but some have a negative direction, i.e. we expect them to decrease (e.g., number of errors/sentence).

outcome_negative <- data.frame(Timing = c(1, 2), Mean = c(.067, .036), SD = c(.251, .187))
# outcome_negative is reduced in ±half between pre and post-test: it is very positive!
# let's just copy it to the target variable before adapting the values
outcome_positive <- outcome_negative

I am computing the effect sizes for all these values, through Standardized Mean Difference (d), but I could not find references on how to manage negative direction effects.

I have thought about:

  • $-\bar{x}$
    • outcome_positive$Mean <- -outcome_negative$Mean
    • What about the standard deviations? $-s$ ? or $s$ ?
  • $\dfrac{1}{\bar{x}}$
    • outcome_positive$Mean <- 1/outcome_negative$Mean
    • What about the standard deviations? $\dfrac{1}{s}$ ? $s$ ?
  • Reversing pre- and post-test values (but it seems even more wrong)
    • outcome_positive$Mean[outcome_positive$Timing == 1] <- outcome_negative$Mean[outcome_negative$Timing == 2] outcome_positive$Mean[outcome_positive$Timing == 2] <- outcome_negative$Mean[outcome_negative$Timing == 1]

Most of all, I wonder what is statistically sound. Don't hesitate to send me to a reference document.

$\endgroup$
0
5
$\begingroup$

In general, you should use $-\bar x$ as the effect size to account for the change in polarity of the outcome, but leave $s$ unchanged.

  • A heuristic argument for leaving $s$ unchanged would be that negative standard errors don't make any sense.
  • A technical argument would be that

$$ \sigma(-x) = \sqrt{\sigma^2(-x)} = \sqrt{(-1)^2\sigma^2(x)} = \sqrt{\sigma^2(x)} = |\sigma(x)| = \sigma(x) $$ according to the basic rules governing variances (i.e. if $c$ is a constant, $\sigma^2(cx) = c^2 \sigma^2(x)$).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.