5
$\begingroup$

Consider the situation in which one should investigate, which treatment or treatments (of A, B, C, D, E, F, G, H) is/are the most effective (highest decrease) and which is/are the worst (lowest decrease, see figure below).

enter image description here

One made an ANOVA/Kruskal-Wallis-like test, which showed a statistically significant difference. Then continued with posthoc pairwise comparisons and summarized the results in the plot below. Non-capital letters a, b, c (the compact letter display, cld) above the box-plots and jittered points of data indicate statistical (in)significance in a concise way: if treatment groups share the same non-capital letter, then the differences between the groups are not statistically significant. E.g., comparing treatments G and H result is insignificant ($p \ge 0,05$) as G and H shares the same letter "e".

Questions:

  1. It's not clear for me: basing on the results, how should I answer the question, which treatment (or group of treatments) is the most effective and which is the least effective?
  2. Is it correct to state that treatments, which share letter "a", are the least effective and the ones, which share "e", are the most effective? Won't it be a misinterpretation of the results as there is no strict boundary between groups of treatments, e.g., treatment G has letter "e" but E has letters "d" and "e", treatment D has "c" and "d" and so on?

For the analysis, I used R and dataset called OrchardSprays.

My question is related to this one but touches different aspects of result interpretation.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ You're unable to tell apart the means of E, F, G, and H, which score the highest, and you're unable to tell apart the means of A and B, which score the lowest. This notation seems to be a summary of all pairwise comparisons, therefore saying that E, F, G, and H are significantly higher than the others (and A and B are significantly lower than the others). $\endgroup$
    – Dave
    Commented Jul 10, 2019 at 18:10
  • $\begingroup$ But D and E do not differ significantly as they share the letter 'd', and the same applies to C and B as they share common letter 'b'? So Are there any general guidelines on how the results like should be interpreted e.g. by APA or other organizations? $\endgroup$
    – GegznaV
    Commented Jul 10, 2019 at 18:19
  • $\begingroup$ This is a good question, and one I've struggled with when writing e.g. journal articles. Usually in these cases I try to avoid saying "highest" or "best", but in cases where I do, I phrase it as "among the highest", and I've always gone with the logic that if we can't say E, G, F, and H are statistically different, then they are each "among the highest". $\endgroup$
    – Sal Mangiafico
    Commented Jan 20, 2020 at 17:16
  • 1
    $\begingroup$ It's also fascinating considering that we can see the individual data points on the plot. Practically speaking, F appears unreliable, and H appears higher than the others and relatively consistent. With the smallish sample size in each group, I might but less faith in the p values, and look more carefully at effect sizes, the spread in each group, and the practical consequences. I know that wasn't exactly the force of the question. $\endgroup$
    – Sal Mangiafico
    Commented Jan 20, 2020 at 17:21
  • $\begingroup$ I like @RonJensen 's thoughts below... One criticism of these compact letter displays is that they give the impression that groups with the same letter are similar in value, when in fact we really want to convey that we don't have good evidence that they are different. (Idea credited to user rvl elsewhere on this site, if I have in fact fairly conveyed the idea). There's also an issue that it treats a p value of, say, 0.05 as a magic cutoff. That being said, I still think compact letter displays are valuable in that they condense a lot of information in an intuitive way. $\endgroup$
    – Sal Mangiafico
    Commented Jan 20, 2020 at 17:37

2 Answers 2

Reset to default
1
$\begingroup$

Seems to me that you are wanting statistical test-based criteria for describing the relationships between the treatment efficacies. Don't do that! Instead, describe the order of size of effect (use the mean and/or the median effects) and leave it there.

The sample sizes are small relative to the variability and inter-treatment effect size differences and so the descriptor of "most effective" for your data might not be reliable for a larger population.

(It is worth noting that all of the treatments appear to be at least minimally effective, and so a dose-response curve for each treatment might show that they are equally effective but differ in potency. Do not discard drug candidates on the basis of a single dose study.)

Improve this answer
$\endgroup$
0
$\begingroup$

I have changed my mind since my comment.

The only two places where there is ambiguity in my mind is if E is tied with F, G, and H, and if B is tied with A. I say no. I am going to talk in absolutes and say that we know something in the next paragraph. That avoids the awkwardness of always saying that we have evidence through some significant p-value.

We know that F, G, and H are contenders for the top spot. We don't know if E is in that group. However, we know that D is not as as good as F, G, and H. Since we cannot find a significant difference between D and E, E should not be regarded as a contender for the top spot. Similarly, we know that A is a contender for the worst performance. We don't know about B. However, we know that C outperformed A. Since we do not have a difference between B and C, B should not be regarded as a contender for the bottom spot.

Consequently, A is the lone group that can be said to have performed the worst, while further testing is needed to tell which is the best of F, G, and H, though these three are the top performers.

Edit: Read the comments.

Improve this answer
$\endgroup$
2
  • $\begingroup$ We are comparing confidence intervals on the means of eight samples. This is the range where the true mean lies in repeated experiments. The 95% confidence intervals of E, F, G, H overlap so we can't know if true means are different or not. They could different, but we don't know if they are. We do know the confidence interval for D does not overlap F, G, H. That is evidence D's true mean is lower than those 3. E is indeterminate, it may be anywhere between the best and the 5th best, this test does not give evidence either way. $\endgroup$
    – Ron Jensen
    Commented Jul 10, 2019 at 19:55
  • $\begingroup$ That was my initial thinking, and I think I've come back around to that. In the same way, B may be anywhere from worst to third-worst. $\endgroup$
    – Dave
    Commented Jul 10, 2019 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.