1
$\begingroup$

My question is about a 2-way ANOVA for an experiment. For simplicity let's say I'm using 3 methods to stimulate cells at 3 different time points, and I want to look at a resulting output variable. Different cells were stimulated at each time point, so independence of observations is maintained. Each time point has 9 - 10 replicates. The data look like this: https://i.imgur.com/oGEagOY.png

I ran a 2-way ANOVA, and found both stimulation method and time to be significant. The multiple comparisons options in Graphpad though are confusing me. I could compare each value to each other value, but this results in irrelevant comparisons being made that decrease significance levels. The comparisons I'm interested in are:

1) The difference in mean output value between each time point. Prism calls this the main row effect.

2)The difference in mean output value between each stimulation method. Prism calls this the main column effect.

3)The difference in output values across stimulation methods within each time point (e.g. comparing day 7 method 1 vs method 2 vs method 3). Prism calls this simple effect within columns.

4)The difference in output values across time points within each stimulation (e.g. comparing stimulation method 1 day 7 vs day 10 vs day 14). Prism calls this a simple effect within rows.

My question is, can I run 4 multiple comparisons using each of these settings, and then use the p-values that are returned?

I would think certainly not, since doing them separately gives a way inflated p-value. So, I tried a similar 2-way ANOVA on both SPSS and Graphpad Prism. These data had no replicates, so I only wanted to compare the columns to each other and the rows to each other. On Prism, I ran the main row effect and main column effect multiple comparisons separately. To my surprise the Prism output matched the SPSS output. But in a 2-way ANOVA post-hoc shouldn't every row mean be compared to not only the other row means but the column means as well?

Sorry for the length. This is my first question so just let me know if I violated any rules and I'll edit accordingly.

Improve this question
$\endgroup$
1
$\begingroup$

Extended Comment: It would be helpful to know P-values for Time, Method, and Time*Method interaction.

Look at Interaction first: It is possible that certain patterns among interactions may affect how differences among Times and differences among Methods are assessed.

A typical way to judge the impact of (significant) interaction is to assess significance of 'orthogonal contrasts'. Considering that you have nine levels of Method (ignoring Method 10, which has only partial data) and three levels of Time, there are $(9-1)(3-1) = 16$ mutually orthogonal contrasts. Ordinarily, that would be more than enough to explore interesting differences.

You may have had some of these contrasts in mind from the start. (For example, you may have expected Methods 1,2,3 to work better if administered early, and other Methods to work better if administered later on.) If you had this in mind before seeing the data, you can look at several of them according to the standards for judging 'pre-chosen' contrasts. Any additional contrasts suggested by the data, should be assessed using a method for 'ad hoc' contrasts (e.g., 'Scheffe's method'). Those methods should keep you from 'false discovery' of effects from artifacts of your particular data.

Then look at comparisons of levels of each main effect: With meaningfully large interaction effects in mind, you can turn to assessing differences among Times and differences among Methods. For each main effect you can keep the 'family error rate' for comparisons among its levels by using a multiple-comparison method such as 'Tukey's HSD'.

Notes: (1) I take the data in your link to be averages of "9 or 10" replications at each 'cell' (combination of levels of Time and Method). Without knowing the variability within cells, it is not possible to judge the level of significance of Interaction. And because the detective work unraveling the meaning of results from a two-factor ANOVA must begin by assessing Interaction, I can have no informed hunch where the suggestions above may lead.

However, in my experience, researchers often tend to design experiments in such a way that interaction effects are not statistically significant or (even if significant) not large enough to be of practical importance. If that is true with your experiment, you might be able to go directly to assessing the significance and importance of the main effects.

(2) For more detail, you can search this site or the Internet for words or phrases I have put in 'single quotes'.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for commenting. p = 0.0004, <0.0001, and 0.9497 for Time, Stimulation Method, and Interaction respectively (so Interaction was non-significant). To clarify, I have 3 levels of Stimulation Method and 3 levels of Time. The data I showed were for Method 1. 9 or 10 different cells were stimulated with each Method at each Time point. So each spreadsheet cell in the image I linked to refers to a different biological cell. I think what you wrote about the family error rate approaches the heart of my question. So b/c the columns and rows are separate families, they can be compared separately? $\endgroup$
    – user219200
    Aug 30 '18 at 21:17
  • $\begingroup$ Then ignore interaction and go on to make separate Tukey HSD comparisons for Time and for Method. One caveat: the P-value for Interaction seems very large. That may just be by chance or may indicate that you have typos in your data, that the residuals from the model are far from normal or not independent (which you should check), that you have not specified the intended model, or even that your software is not working properly, (Old rule among applied statisticians: "If the P-value is tiny, suspect $H_0;$ if the P-value is huge, suspect the model or the computations.") $\endgroup$
    – BruceET
    Aug 30 '18 at 21:46
  • $\begingroup$ Interesting comment about the too-large interaction p-value, I hadn't heard that. I will be checking the assumptions of course. Could I do separate Tukey HSD comparisons for Time, Method, and the effects I mention as 3) and 4) in my original post as well? Do 3) and 4) count as separate families too? $\endgroup$
    – user219200
    Aug 31 '18 at 5:24
  • $\begingroup$ Unless the comparisons in 3) & 4) where chosen before you saw data --and with convincing reasons-- I'd wonder why these particular pairs among ${27 \choose 2}=531$ possible choices. I don't see how Tukey HSD applies to these unless you do separate ANOVAs for the relevant rows / columns. Would use Bonferroni. $\endgroup$
    – BruceET
    Aug 31 '18 at 7:03
  • $\begingroup$ I see. They were chosen before I saw the data, but yes I have considered separating the data into different ANOVAs. I think I'll look into Tukey's HSD and what constitutes an independent family. I've marked my question as answered, thanks for the help. $\endgroup$
    – user219200
    Aug 31 '18 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.