Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I run an experiment to compare the performance of two algorithms. The design of my experiment is paired comparisons. I'm reporting my results in this way:

There were no outliers in mean differences data as assessed by inspection of boxplot. The assumption of normality was not violated as assessed by skewness of 0.2276 (SE = 0.4405) and kurtosis of -0.2766 (SE = 0.8583). The performance was higher in algorithm A (M = 0.3876, SD = 0.3138) than in algorithm B (M = 0.2241, SD = 0.3476), a statically significant mean increase of 0.1635, 95% CI [0.0393, 0.2877], t(27) = 2.7007, p = 0.0118, d = 0.4938, 95% CI for d [-0.0501, 1.0378].

Note that the 95% confidence interval for mean difference does not include zero but the 95% confidence interval for Cohen'd includes zero. I was about to conclude that algorithm A has better performance than algorithm B with statistical significance and medium effect size, but I got confused about how to interpret the confidence interval for Cohen's d.

What can I say about the effect size with this data?

Below is my data and how I compute the values using R.

Thanks for your attention.

a = c(0.40000000, 0.44011976, 0.72727273, 0.50000000, 0.00000000, 0.07692308, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.50000000, 0.91666667, 0.19354839, 0.74883721, 0.50000000, 0.50000000, 0.55000000, 0.17142857, 0.50000000, 0.51351351, 0.68000000, 0.85714286, 0.03703704, 0.05454545, 0.54219949, 0.44444444, 0.00000000, 0.00000000)
b = c(0.00000000, 0.54491018, 0.72727273, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 0.00000000, 1.00000000, 0.00000000, 0.00000000, 0.00000000, 0.33953488, 0.00000000, 0.00000000, 0.00000000, 0.48571429, 0.00000000, 0.83783784, 0.80000000, 0.57142857, 0.00000000, 0.00000000, 0.06393862, 0.90476190, 0.00000000, 0.00000000)
mean(a)
sd(a)
mean(b)
sd(b)
t.test(a, b, paired=TRUE)

library(compute.es)
mes(mean(a), mean(b), sd(a), sd(b), length(a), length(b), dig=4)
share|improve this question

2 Answers 2

up vote 3 down vote accepted

There are analogous effect size measures to Cohen's for paired data, sometimes called the "standardized mean change" or "standardized mean gain". This is computed with $$d = \frac{\bar{x}_1 - \bar{x}_2}{SD_D} = \frac{\bar{x}_D}{SD_D},$$ where $\bar{x}_1$ is the mean at time 1 (or under condition 1), $\bar{x}_2$ is the mean at time 2 (or under condition 2), $\bar{x}_D$ is the mean of the change/differences scores, and $SD_D$ is the standard deviation of the change/differences scores.

This is the standardized mean change using "change score standardization". There is also the standardized mean change using "raw score standarization", but the former more directly relates to your use of the dependent samples t-test.

You can use the metafor package to compute this (and the corresponding CI):

summary(escalc(measure="SMCC", m1i=mean(a), sd1i=sd(a), m2i=mean(b), sd2i=sd(b), ni=length(a), ri=cor(a,b)))

yields:

      yi     vi    sei     zi  ci.lb  ci.ub
1 0.4961 0.0401 0.2003 2.4769 0.1035 0.8886

So, now the CI doesn't include 0 anymore, which is consistent with the results from the t-test. (Note: the value under yi is the d-value above, but after using a slight bias correction).

Some references if you want to read more about this:

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological Methods, 7, 105–125.

Viechtbauer, W. (2007). Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design. Journal of Educational and Behavioral Statistics, 32, 39-60.

Update: Getting the exact CI for d.

In rare cases, it can happen that the results of the t-test (and the CI for the mean difference) yields a different conclusion than the CI for d obtained above (i.e., the CI for the mean difference includes the value 0, while the CI for d does not, or vice-versa). This is due to the fact that the CI for d is based on an asymptotic approximation using the normal distribution.

One can compute an exact CI for the standardized mean change, but this requires iterative methods (see Viechtbauer, 2007, and the references given therein). The advantage of the exact CI is that it will always agree 100% with the results from the t-test and the CI for the mean difference in its conclusion.

Instead of letting the computer do the iterative work for us (which can be done in a few lines of code), one can also just do this manually by trial and error. For the data given in http://pastebin.com/12J7UghC, the bounds of the exact CI for d can be obtained with:

tval <- t.test(a, b, paired=TRUE)$statistic
pt(tval, df=length(a)-1, ncp=-0.00265265 * sqrt(length(a)), lower.tail=TRUE)
pt(tval, df=length(a)-1, ncp=-0.77193310 * sqrt(length(a)), lower.tail=FALSE)

Essentially, we just need to find those two values of the non-centrality parameter of the t-distribution, so that the observed t-value cuts off .025 in the lower and upper tails of the distribution. With a bit of trial and error (and starting with the CI bounds obtained earlier), we find the exact 95% CI for d is $(-0.003, -0.772)$. And now things are consistent again: The t-test rejects (just barely, with $p=.048$), the CI for the mean difference excludes 0 (just barely), and the CI for d exclude 0 (just barely).

share|improve this answer
    
Thank you, Wolfgang! –  Alan Moraes Feb 18 at 22:54
    
You are welcome. Aside from accepting an answer (if you find that it answers your question), you should also consider upvoting answers that are useful. For example, Peter's response was useful in pointing out the source of the inconsistency. –  Wolfgang Feb 18 at 23:02
    
I'm sorry. I didn't know that because this was my first question here. Upvoted both answers. I still have data that shows inconsistency between the two confidence intervals, even using the correct method: 95% CI for mean difference [-0.222368907, -0.000822525] and 95% CI for Cohen's d [-0.7632, 0.0038]. I guess this is because -0.000822525 is too close to zero and I should not reject the null hypothesis based on CI for Cohen's d. Data is available here: pastebin.com/12J7UghC Thanks again! –  Alan Moraes Feb 18 at 23:19
    
No problem. Just wanted to bring this to your attention. And I'll update my answer to deal with the issue you are encountering now. –  Wolfgang Feb 19 at 0:10
    
Thank you for the patience, Professor Viechtbauer. Your answer was clear and to the point. I'll look the cited references. Note: I could reproduce your results using pt(tval$statistic, ...) instead of pt(tval, ...). –  Alan Moraes Feb 19 at 3:40

mes does not account for the paired-ness of the data, so, it is a different test; usually, paired tests are more powerful, so it is not surprising that the paired test was significant and the unpaired was not.

share|improve this answer
    
Hi, Peter, thanks for your reply. I read somewhere that the correct way to calculate Cohen's d is using the result of t-test for unpaired data. Whenever I find the reference I'll post here. –  Alan Moraes Feb 18 at 21:19
1  
Here is the reference: Dunlap, W. P.; Cortina, J. M.; Vaslow, J. B. & Burke, M. J. Meta-Analysis of Experiments With Matched Groups or Repeated Measures Designs Psychological Methods, 1996, 1, 170-177. dropbox.com/s/1bbz20lehbff1cc/… –  Alan Moraes Feb 18 at 21:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.