1
$\begingroup$

I have been trying to work through this idea for a few days now, and I think I am just really confusing myself at this point so thought I could hopefully get some help.

The basic idea is, I have 1 control / reference group and 3 treatment groups. I do not care about any comparisons between any treatment groups to each other, I only care if there is a difference between any of the treatment groups and control. So, as I understand it, the overall F test from anova is not useful here, right? As, it will tell me if ANY group mean is different from any other.

As far as I am aware, there is not like stat that will just tell me if there are any groups that different between control and any treatment. Like an overall F stat. I know there is Dunnett's test, but that just gives the individual contrasts. Is there something like an anova but the null is just comparing the the difference between the reference group and any of the treatment groups (not the mean of the groups, I know I can do that but that isn't the same thing, right?)?

And, the confusion now that I am having is, does this even make sense? What I mean is, it's not really theoretically possible for 2 treatment groups to be different from each other and then there not to be a difference from the control group, so an overall F test from anova DOES make sense, right? Obviously in practice, sometimes this happens, for example I have a dataset where I get the following from my anova and Tukey post-hoc tests:

Factor with 4 levels (one control reference and 3 treatments)
Overall F test IS significant
C vs T1 -- not sig
C vs T2 -- not sig
C vs T3 -- not sig
T1 vs T2 -- sig
T1 vs T3 -- sig
T2 vs T3 -- not sig

I know this is kind of a different question, but also goes along with the overall F test idea of actually making sense, I think? But, how is the above possible? Power? The idea that two groups not being statistically significantly different doesn't mean they are the "same"? I guess... I think I am just getting very confused. And any help would be appreciated!

To summarize:

  1. Is there an overall F test for a Dunnett's test type comparisons? (So something that tells you if any of the control vs treatment comparisons are significant, not caring about the treatment to treatment ones like anova would) Or does overall F test from anova work for this, and if so, why?
  2. How is that anova vs Tukey post-hoc testing scenario possible?
$\endgroup$

2 Answers 2

1
$\begingroup$

I say the overall test you want is the usual F-test. You say you do not care if two treatment groups have different means. However, if two treatment groups have different means, then at least one of them must have a mean different from that of the control group. After all, if $\mu_{T_i}\ne\mu_{T_j}$, then $\mu_C = \mu_{T_i}$ and $\mu_C = \mu_{T_j}$ cannot both be true, because then $\mu_{T_i}=\mu_{T_j}$.

($\mu_C$ is the control mean. $\mu_{T_i}$ and $\mu_{T_j}$ are the means of distinct treatment groups $i$ and $j$.)

$\endgroup$
0
$\begingroup$

For your second question, yes, it is possible and it illustrates the issues with relying to "sig." vs. "not sig." This is not something to rely on. But don't take my word for it, Andrew Gelman and Hal Stern have a paper about it. (You can tell it is possible because it happened). Also, you are correct that two groups not being significantly different does not mean they are the same.

The Difference Between "Significant" and "Not Significant" is not, Itself, Statistically Significant

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.