7
$\begingroup$

Update:

I haven't gotten any answers, so I am simplifying my question. Say I measure test scores of $100$ people in one of two conditions. Each test has $10$ items. On each item, each person can receive a score of $0-10$. Each person's test score is the average across items. My question is, is the effect size of the condition variable better calculated as the mean score difference between the two conditions divided by a standard deviation calculated from:

  1. the $100$ test scores, or
  2. the $10000$ item scores?

I realize there are better effect size measures than the two presented above, and you're welcome to offer a suggestion, but what I really want to know is if it is defensible philosophically to ignore the within-person variation simply because "the" relevant measure is each person's average score across items (e.g., on the SAT, persons only see their total test score, not their performance on individual items).

I would never ignore variation in most analyses, but my colleague insists that because all that matters is the total test score, the effect size measure should ignore within-person variability.

Original question:

I have $10$ scores for each of $n$ subjects in each of $2$ time points. I want to compute an effect size for the increase in scores from the first to the second time point. My colleagues have previously reported $(M_2 – M_1) / \sigma_{pooled}$ as the standardized gain in scores, where $M_j$ was the sample mean for time point $j$, and $\sigma_{pooled}$ was a pooled estimate calculated from the sample standard deviations of the $n$ mean subject scores for each time point; that is, for each time point, they took the mean score for each subject and then computed an SD estimate based on those means and then pooled those SD estimates.

This measure of a standardized score gain does not fit with my intuition because it ignores both within-subject variability and the correlation of scores across time points but within subjects. I am only concerned with the first issue at the moment.

  1. In this context, should a measure of effect size take into account within-subject variance or not, and why or why not?

  2. If it should, what are some sensible measures of the effect size of the increase in scores from the first to the second time point? Preferably it would be on the metric of standardized scores or standardized difference scores for ease of interpretation, but other types of effect sizes are okay too. Citations are welcome because I am not familiar with this area, but are not necessary.

Finally, below is reproducible R output and code for the model I am using for the data. This might or might not be relevant to the computation of an effect size, but I am including it for completeness. It treats the effect of time (condition) as a fixed effect, and estimates two correlated subject-level random intercept variances – one for the first time point and one for the second time point.

library(lme4)
set.seed(5000)
ncondition <- 2
nsubj <- 200
ntrial <- 10
sd.resid <- 5
sd.cond.1 <- 2
sd.cond.2 <- 1
cor.1.2 <- .7
cov.1.2 <- cor.1.2 * sd.cond.1 * sd.cond.2
Sigma <- matrix(c(sd.cond.1^2, cov.1.2, cov.1.2, sd.cond.2^2), 2, 2)
Condition <- rep(rep(1:ncondition, each = ntrial), times = nsubj)
Condition.effect <- rep(rep(c(5.0,10.0), each = ntrial), 
                            times = nsubj)
Subject <- rep(1:nsubj, each = ncondition*ntrial)
Subject.effect <- rep(matrix(t(mvrnorm(n = nsubj, c(0, 0), Sigma))), 
                      each = ntrials)
Trial <- rep(rep(1:ntrial, times = ncondition), times = nsubj)
Error <- rnorm(n = nsubj*ncondition*ntrials, mean = 0, sd = sd.resid)
Y <- Condition.effect + Subject.effect + Error
datn <- data.frame(Condition, Subject, Trial, Condition.effect, 
                   Subject.effect, Error, Y)
datn$Condition <- factor(datn$Condition)
datn$Subject <- factor(datn$Subject)
datn$Trial <- factor(datn$Trial)
(mod <- lmer(Y ~ -1 + Condition + (0 + Condition | Subject), 
             data=datn))

# Linear mixed model fit by REML 
# Formula: Y ~ -1 + Condition + (0 + Condition | Subject) 
#    Data: datn 
#    AIC   BIC logLik deviance REMLdev
#  24596 24634 -12292    24580   24584
# Random effects:
#  Groups   Name       Variance Std.Dev. Corr  
# Subject  Condition1  4.7814  2.1866         
#          Condition2  1.1729  1.0830   0.682 
# Residual            25.5615  5.0558         
# Number of obs: 4000, groups: Subject, 200
# Fixed effects:
#           Estimate Std. Error t value
# Condition1   4.8272     0.1915   25.20
# Condition2  10.1398     0.1365   74.27
Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ see also this question on effect size for repeated measures t-test stats.stackexchange.com/questions/11653/… $\endgroup$
    – Jeromy Anglim
    Commented Oct 31, 2011 at 8:08
  • $\begingroup$ @Jeromy, thanks -- the thread was helpful, but my first question is whether it is defensible to average within subjects and then compute a between-subjects sample SD. This makes the effect size double what it would be if I simply used a pooled sample SD estimate treating each data point as if it came from a different subject. $\endgroup$
    – lockedoff
    Commented Oct 31, 2011 at 14:34

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.