2
$\begingroup$

If I have two factors - 'A' and 'B'. Each factor has three levels (A1,A2,A3,B1,B2,B3). If I have already performed t-test for each pair such as (A1,A2). All result suggest no significant difference. Do I need to perform ANOVA and even post-hoc test? From my opinion, the correction of post-hoc is used to minimize type I error. Therefore, it ask for higher differences to show significance. As a result, if t test show no significant difference, the post-hoc test will also show no significance.

$\endgroup$
2
  • 1
    $\begingroup$ It is possible for the collection of paired t-tests to all be n.s., but the overall ANOVA for that factor could be statistically significant. Also, there is a chance there is an interaction effect...in other words, ({A1+B1},{A2+B2}) could be statistically significant. $\endgroup$
    – Gregg H
    Commented Jun 15, 2023 at 19:02
  • $\begingroup$ @GreggH is correct. Especially if A and B are related to each other. Also, don't rely on significance too much when making decisions. about what analysis to do. $\endgroup$
    – Peter Flom
    Commented Jun 15, 2023 at 19:43

2 Answers 2

6
$\begingroup$

First, I would avoid using the word "significance", which tends to obscure clear thinking about exactly what questions you want a statistical analysis to answer.

If you run a two-way ANOVA with appropriate followup testing, you'll obtain the 95% confidence intervals for all the differences (between means) that you are interested in. These calculations will account for multiple comparisons, so the 95% confidence level applies to the whole family of comparisons, not to each one individually. They will tell you how precisely you know the difference between means.

In many situations, it makes more sense to think about statistics as quantifying uncertainty, rather than making binary decisions about statistical significance.

$\endgroup$
4
$\begingroup$

Your question is quite specific, but you are likely to get answers that are only obliquely aligned to it. The direct answer is short (see Gregg H's comment). This will, of course, be one of those oblique answers...

What are your inferential objectives? Statistical comparisons of A1 to A2 and A2 to A3 and A1 to B1 and A1 to B2 et cetera are unlikely to be equally helpful for the types of inference that would typically be desired. Do you want to know about differences among the treatment levels or about the differences between the factors? Are you doing an exploratory study (probably) or is it a study designed to obtain a definitive answer to a question?

As you have three levels of treatment, the pharmacologist within me is shouting that you have performed a dose-response study. Plot the data and inspect it for a dose-response relationship within the A and B categories. Assuming that A and B both were effective, then you would also be able to see whether there is an obvious difference in the potencies of A and B.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.