1
$\begingroup$

I have three paired groups, let's call them A, B and C.

I want to check if A's mean is statistically better than B and if A's mean is better than C. I don't care how B and C interact.

I believe I should run paired one sided t tests. But it seems like if I want to have 95% confidence, overall, I can't use 95% for each test.

Am I correct in wanting to use one sided t tests? If so, What confidence value should I use to compare against?

Secondly suppose I have 5 separate, completely independent frameworks; within each framework, I have paired values for A, B, and C. Can I run the above one sided t tests 5 times (once for each framework)? Or must I do something more complicated.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Since you do only two tests, divide the $\alpha$ level per test in two. That's a Bonferroni correction. You don't need to correct for a third possible test that doesn't interest you. No need to divide by three.

Do two sided tests. One sided tests should only be done if it is logically impossible for the outcome to be on the other side (i.e. you are testing a quantity that can only ever be positive against the null hypothesis that it is 0).

Don't use your intuitions to justify one sided tests. Everybody will think you made the theory up after seeing the direction of the result and you wanted to have more significant results.

Improve this answer
$\endgroup$
7
  • $\begingroup$ This is helpful thanks. I will switch over to two sided tests. If I have 5 separate frameworks for A,B,C, should I divide by 10 instead of 2? Also, if I have two overarching universes I am testings each of the 5 separate frameworks, should I divide by 20? $\endgroup$
    – user176635
    Commented Sep 10, 2017 at 0:33
  • $\begingroup$ Depends what you mean by frameworks and universes. Anyway, if you find a way to join those tests rather than to divide them, your statistical power will increase not decrease. But if you have 20 tests and declare discovery as soon as one of the 20 is significant, you need to correct for 20 tests. Do a Holm correction if you have more than 2 tests, it is better than the Bonferroni correction. $\endgroup$
    – David Ernst
    Commented Sep 10, 2017 at 0:39
  • 2
    $\begingroup$ On what reasoning is this based: "One sided tests should only be done if it is logically impossible for the outcome to be on the other side" ....? $\endgroup$
    – Glen_b
    Commented Sep 10, 2017 at 6:18
  • 1
    $\begingroup$ Would you recommend paired t-tests over a repeated measures ANOVA? $\endgroup$
    – user176635
    Commented Sep 10, 2017 at 16:08
  • 1
    $\begingroup$ There is an accept answer option you know :) and upvotes too $\endgroup$
    – David Ernst
    Commented Sep 11, 2017 at 2:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.