10
$\begingroup$

Background

I am conducting a meta-analysis that includes previously published data. Often, differences between treatments are reported with P-values, least significant differences (LSD), and other statistics but provide no direct estimate of the variance.

In the context of the model that I am using, an overestimate of variance is okay.

Problem

Here is a list of transformations to $SE$ where $SE=\sqrt{MSE/n}$ (Saville 2003) that I am considering, feedback appreciated; below, I assume that $\alpha=0.05$ so $1-^{\alpha}/_2=0.975$ and variables are normally distributed unless otherwise stated:

Questions:

  1. given $P$, $n$, and treatment means $\bar X_1$ and $\bar X_2$ $$SE=\frac{\bar X_1-\bar X_2}{t_{(1-\frac{P}{2},2n-2)}\sqrt{2/n}}$$

  2. given LSD (Rosenberg 2004), $\alpha$, $n$, $b$ where $b$ is number of blocks, and $n=b$ by default for RCBD $$SE = \frac{LSD}{t_{(0.975,n)}\sqrt{2bn}}$$

  3. given MSD (minimum significant difference) (Wang 2000), $n$, $\alpha$, df = $2n-2$ $$SE = \frac{MSD}{t_{(0.975, 2n-2)}\sqrt{2}}$$

  4. given a 95% Confidence Interval (Saville 2003) (measured from mean to upper or lower confidence limit), $\alpha$, and $n$ $$SE = \frac{CI}{t_{(\alpha/2,n)}}$$

  5. given Tukey's HSD, $n$, where $q$ is the 'studentized range statistic', $$SE = \frac{HSD}{q_{(0.975,n)}}$$

An R function to encapsulate these equations:

  1. Example Data:

    data <- data.frame(Y=rep(1,5), 
                       stat=rep(1,5), 
                       n=rep(4,5), 
                       statname=c('SD', 'MSE', 'LSD', 'HSD', 'MSD') 
    
  2. Example Use:

    transformstats(data)    
    
  3. The transformstats function:

    transformstats <- function(data) {
      ## Transformation of stats to SE
      ## transform SD to SE
      if ("SD" %in% data$statname) {
        sdi <- which(data$statname == "SD")
        data$stat[sdi] <- data$stat[sdi] / sqrt(data$n[sdi])
        data$statname[sdi] <- "SE"
          }
      ## transform MSE to SE
      if ("MSE" %in% data$statname) {
        msei <- which(data$statname == "MSE")
        data$stat[msei] <- sqrt (data$stat[msei]/data$n[msei])
        data$statname[msei] <- "SE"
      }
      ## 95%CI measured from mean to upper or lower CI
      ## SE = CI/t
      if ("95%CI" %in% data$statname) {
        cii <- which(data$statname == '95%CI')
        data$stat[cii] <- data$stat[cii]/qt(0.975,data$n[cii])
        data$statname[cii] <- "SE"
      }
      ## Fisher's Least Significant Difference (LSD)
      ## conservatively assume no within block replication
      if ("LSD" %in% data$statname) {
        lsdi <- which(data$statname == "LSD")
        data$stat[lsdi] <- data$stat[lsdi] / (qt(0.975,data$n[lsdi]) * sqrt( (2 * data$n[lsdi])))
        data$statname[lsdi] <- "SE"
      }
      ## Tukey's Honestly Significant Difference (HSD),
      ## conservatively assuming 3 groups being tested so df =2
      if ("HSD" %in% data$statname) {
        hsdi <- which(data$statname == "HSD" & data$n > 1)
        data$stat[hsdi] <- data$stat[hsdi] / (qtukey(0.975, data$n[lsdi], df = 2))
        data$statname[hsdi] <- "SE"
      }              
      ## MSD Minimum Squared Difference
      ## MSD = t_{\alpha/2, 2n-2}*SD*sqrt(2/n)
      ## SE  = MSD*n/(t*sqrt(2))
      if ("MSD" %in% data$statname) {
        msdi <- which(data$statname == "MSD")
        data$stat[msdi] <- data$stat[msdi] * data$n[msdi] / (qt(0.975,2*data$n[lsdi]-2)*sqrt(2))
        data$statname[msdi] <- "SE"
      }
      if (FALSE %in% c('SE','none') %in% data$statname) {
        print(paste(trait, ': ERROR!!! data contains untransformed statistics'))
      }
      return(data)
    }
    

References

Saville 2003Can J. Exptl Psych. (pdf)

Rosenberg et al 2004 (link)

Wang et al. 2000 Env. Tox. and Chem 19(1):113-117 (link)

$\endgroup$
  • $\begingroup$ I am not sure if most CIs are really computed via t-values or rather via z-values. However, on bigger ns (> 30) this shouldn't make much of a difference. $\endgroup$ – Henrik Sep 23 '10 at 16:45
  • $\begingroup$ @Henrik for small $n$, the t-statistic is appropriate, and as you said, as $n$ increases, t approximates Z. See also math.stackexchange.com/q/23246/3733 $\endgroup$ – David LeBauer Feb 23 '11 at 19:14
7
$\begingroup$

Your LSD equation looks fine. If you want to get back to variance and you have a summary statistic that says something about variability or significance of an effect then you can almost always get back to variance—-you just need to know the formula. For example, in your equation for LSD you want to solve for MSE, MSE = (LSD/t_)^2 / 2 * b

$\endgroup$
  • $\begingroup$ For MSD, if MSD = t_{alpha,2n-2}*sdsqrt(2/n), is SE = MSDn/(t_{alpha,n}*sqrt(2)) correct? $\endgroup$ – David LeBauer Sep 21 '10 at 16:35
7
$\begingroup$

I can only agree with John. Furthermore, perhaps this paper by David Saville helps you with some formula to recalculate variability measures from LSDs et al.:
Saville D.J. (2003). Basic statistics and the inconsistency of multiple comparison procedures. Canadian Journal of Experimental Psychology, 57, 167–175

UPDATE:
If you are looking for more formulas to convert between various effect sizes, books on meta-analysis should provide a lot of these. However, I am not an expert in this area and can't recommend one.
But, I remember that the book by Rosenthal and Rosnow once helped with some formula:
Essentials of Behavioral Research: Methods and Data Analysis
Furthermore, I have heard a lot of good things about the formulas in this book by Rosenthal, Rosnow & Rubin (although I have never used it):
Contrasts and Effect Sizes in Behavioral Research: A Correlational Approach (You should definitely give it a try if a nearby library has it).

If this is not enough, perhaps ask another question on literature for converting effect sizes for meta-analyses. Perhaps someone more into meta-analysis has more grounded recommendations.

$\endgroup$
0
$\begingroup$

You may consider trying the R package compute.es. There are several functions for deriving effect size estimates and the variance of the effect size.

$\endgroup$
  • $\begingroup$ that is a nice package that you have written, but I am interested in estimating the sample SE, and these functions appear to give variance estimates for the meta analysis effect sizes, whereas I would like to infer the variance of the population (e.g. scaled to the original data). Could you provide an example of how the functions in the compute.es package could be used to replicate the equations and function that I have written above? $\endgroup$ – David LeBauer Mar 17 '11 at 2:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.