12
$\begingroup$

The image below is from this article in Psychological Science. A colleague pointed out two unusual things about it:

  1. According to the caption, the error bars show "±2.04 standard errors, the 95% confidence interval." I've only ever seen ±1.96 SE used for the 95% CI, and I can't find anything about 2.04 SE being used for any purpose. Does 2.04 SE have some accepted meaning?
  2. The text states that planned pairwise comparisons found significant differences for mean startle magnitude in error vs. correct predictable trials (t(30)=2.51, p<.01) and error vs. correct unpredictable trials (t(30)=2.61, p<.01) (the omnibus F test was also significant at p<.05). However, the graph shows the error bars for all three conditions overlapping substantially. If the ±2.04 SE intervals overlap, how can the values be significantly different at p<.05? The overlap is large enough that I'm assuming that the ±1.96 SE intervals also overlap.

bar graph showing 2.04 SE error bars

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Great answers. I would like to stress that (as whuber already pointed out) comparing 95% confidence intervals is not the same as performing statistical tests at the significance level 0.05. There are of course papers dealing with this. If confidence intervals are the only statistics available Payton et al suggest to use 85% intervals for significance level 0.05 for Gaussian data. They follow up their work here. $\endgroup$
    – Martin Berglund
    Commented Jul 5, 2012 at 11:37
  • 1
    $\begingroup$ Thanks, @Martin. To close the loop: although I haven't looked at the Payton et al paper, the basis for 85% is clear: the z-value corresponding to 84%, when squared, equals $2$; adding two of these gives $4$; its square root is $2$, which is pretty much the z-value corresponding to a 95% interval. I suppose Payton rounded 84% to 85%. In other words, their recommendation (however it was derived) can be explained by the same analysis I provided. $\endgroup$
    – whuber ♦
    Commented Jul 5, 2012 at 13:22
  • $\begingroup$ @MartinBerglund and whuber Came across your answers when wondering whether my independent calculation of 83.4% confidence intervals for performing statistical tests at 0.05 level was original - evidently not! Thanks for the paper reference, very helpful. $\endgroup$
    – tristan
    Commented Feb 27, 2013 at 11:40

2 Answers 2

Reset to default
12
$\begingroup$

  1. $2.04$ is the multiplier to use with a Student t distribution with 31 degrees of freedom. The quotations suggest $30$ degrees of freedom is appropriate, in which case the correct multiplier is $2.042272 \approx 2.04$.

  2. Means are compared in terms of standard errors. The standard error is typically $1/\sqrt{n}$ times the standard deviation, where $n$ (presumably around $30+1=31$ here) is the sample size. If the caption is correct in calling these bars the "standard errors," then the standard deviations must be at least $\sqrt{31} \approx 5.5$ times greater than the values of approximately $6$ as shown. A dataset of $31$ positive values with a standard deviation of $6 \times 5.5 = 33$ and a mean between $14$ and $18$ would have to have most values near $0$ and a small number of whopping big values, which seems quite unlikely. (If this were so, then the entire analysis based on Student t statistics would be invalid anyway.) We should conclude that the figure likely shows standard deviations, not standard errors.

  3. Comparisons of means are not based on overlap (or lack thereof) of confidence intervals. Two 95% CIs can overlap, yet can still indicate highly significant differences. The reason is that the standard error of the difference in (independent) means is, at least approximately, the square root of the sum of squares of the standard errors of the means. For example, if the standard error of a mean of $14$ equals $1$ and the standard error of a mean of $17$ equals $1$, then the CI of the first mean (using a multiple of $2.04$) will extend from $11.92$ to $16.08$ and the CI of the second will extend from $14.92$ to $19.03$, with substantial overlap. Nevertheless the SE of the difference will equal $\sqrt{1^2+1^2}\approx 1.41$. The difference of means, $17-14=3$, is greater than $2.04$ times this value: it is significant.

  4. These are pairwise comparisons. The individual values can exhibit a lot of variability while their differences might be highly consistent. For instance, a set of pairs like $(14,14.01)$, $(15,15.01)$, $(16,16.01)$, $(17,17.01)$, etc., exhibits variation in each component, but the differences are consistently $0.01$. Although this difference is small compared to either component, its consistency shows it is statistically significant.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Many thanks. The article doesn't state anywhere that the post-hoc tests were paired comparisons between each participant's responses on the two types of trials, and so I jumped to the conclusion they were treating it as a between-subjects comparison (even though that would be less appropriate and less powerful). I think you must be right, and they were doing the more sensitive (and more difficult to graph) test. As for point #3, my only response is that I clearly need to relearn some statistics... $\endgroup$
    – octern
    Commented Jul 5, 2012 at 2:23
  • $\begingroup$ I was picking up on a phrase in your question, "planned pairwise comparisons." The rest of the results you quote, though, suggest they were not pairwise comparisons, but more likely came from a calculation similar to that in point #3 of my answer. $\endgroup$
    – whuber ♦
    Commented Jul 5, 2012 at 2:25
  • $\begingroup$ What I meant by that was that they were doing post-hoc tests comparing two of the three conditions to each other directly, rather than doing an omnibus test that compared all 3 conditions. Sorry about the confusion. But now that I look at it, I think you were correct anyway. The way they report the omnibus test statistic (F(2,60)=5.64, p<.05) implies that it was a repeated-measures test, and so the post hoc tests likely were as well. $\endgroup$
    – octern
    Commented Jul 5, 2012 at 2:35
  • $\begingroup$ Thank you for your great answer. "The reason is that the standard error of the difference in (independent) means is, at least approximately, the square root of the sum of squares of the standard errors of the means." I am looking for references, which discuss this but could not find any. I would appreciate some guidance in this regard. Maybe someone could help me out? $\endgroup$
    – Johannes
    Commented Nov 8, 2012 at 17:36
  • $\begingroup$ @Johannes The square of the SE is proportional to the variance of the sample mean. (The constant of proportionality depends on one's definition and may vary slightly with sample size.) Independence implies the variance of the sampling distribution of the difference of means is the sum of squares of the SE's. $\endgroup$
    – whuber ♦
    Commented Nov 8, 2012 at 17:51
4
$\begingroup$

Part of the confusion here is the confusing representation of the data. It appears to be a repeated measures design yet the error bars are confidence intervals of how well the true mean value was estimated. A primary purpose of repeated measures is to avoid collecting enough data to get a quality estimate of the raw mean value. Therefore error bars such as those presented really bear almost no relation to the story being told. The value of critical interest is the effect. With the purpose of graphs being to highlight the main point of the story, graphing the effects, and their confidence intervals, would have been more appropriate.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you! I was struggling a bit to express why the graph seemed like it didn't represent the analysis. $\endgroup$
    – octern
    Commented Jul 5, 2012 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.