# Different results for type I and type II ANOVA

I am performing an ANOVA for a research question. I want to to test whether "Grade", "Nim" and "Tim" has a significant influence on "value". My question is regarding the different types for SS.

If I use type I, "Grade" and "Tim" are significant while "Nim" is not. This was to be expected if I evaluate the differences in means. However, if I use tpye 2 ANOVA, than also "Nim" is highly significant even if the means are almost equal. How can it be and how to interpret this?

The vector in "values" is actually dependent on another variable called (x) and I vary x in a certain range an test significant differnce in each obtained set of data. So this is just one example of a vector "value". I observed that even if the group means by Nim flipped in order, I dont obtain a p-value larger 0.05.

library(rstatix)

data <- data.frame(Grade=c(0, 0, 0, 1, 1, 1),
Nim =c(0, 1, 0, 0, 1, 0),
Tim = c(0, 0, 1, 0, 0, 1),
value=c(44.68, 59.53, 77.08, 39.17, 55.92, 70.27))
#Type I ANOVA
data %>% anova_test(value ~ Grade + Nim + Tim, type=1)

#Type II ANOVA
data %>% anova_test(value ~ Grade + Nim + Tim, type=2)

#means
data %>%
summarize(mean = mean(value))

data %>%
group_by(Nim) %>%
summarize(mean = mean(value))

data %>%
group_by(Tim) %>%
summarize(mean = mean(value))

• Are you familiar with what the types mean? See this. I don't understand your last paragraph. Why are you not simply investigating the relationship with x? Also, you might need to adjust p-values for multiple comparisons. I suspect you should consult a statistician. Feb 14, 2023 at 7:07
• Yes, i know what the types mean. I just dont understand how type 2 or type 3 can yield a highly significant effect of "Nim" even though the means are equal. Regarding the last paragraph: Lets imagine "values" were evaluated at different times x. Now i want to test, for which x the independent values had a significant influence on "values". Now i observed that for grouping by "Nim" for some x mean(group1) > mean(group2), while for other this relation is flipped. However, p values were always <<0.05. Feb 14, 2023 at 7:33
• Then you should include time in your model. Feb 14, 2023 at 7:50
• The results of a Type I ANOVA depend on the order of terms, because it adds varaibles sequentially. If you exchange the terms Tim and Nim, e.g., the type I ANOVA will also report Nim as significant. This is not the case for a type III (or type II) ANOVA, because it drops each term separately. A workaround for the type I problem is to try out all permutations and average the results, a method known as "dominance analysis". This can be interpreted as a measure for variable importance (see the R package relaimpo for details). Feb 14, 2023 at 9:31
• Thanks for the comments. I understand the differences in the types. What i don't understand is, how ANOVA can yield a significance for the variable "Nim" even though the means of the groups are pretty much identical. Feb 14, 2023 at 16:40

There are two issues with your analysis. (a) You are trying to interpret the statistical significance of predictors rather than understand their effect on the outcome. In other words, you are looking at p-values rather than at a scatter plot of your data. (b) You are comparing the unadjusted group means for each predictor separately while the ANOVA sums of squares are adjusted.

Since you know the difference between the type I and type II ANOVA, I won't explain it. If you need a review, read this great answer: How to interpret type I, type II, and type III ANOVA and MANOVA?

Now, the model is Y ~ Grade + Nim + Tim and you are investigating the difference between the type I and type II sums of squares for the predictor in the middle, Nim. With respect to Nim, the type I ANOVA means to adjust for Grade first and then compute the sum of squares of Nim. And the type II ANOVA means to adjust for both Grade and Tim and then compute the sum of squares for Nim.

Let's illustrate the difference between adjusting for Grade alone and adjusting for both Grade and Tim. Since there are no interactions in the model, "adjusting" is equivalent to fitting a regression and taking the residuals. The left panel shows the raw measurements; the middle panel corresponds to Y ~ Grade; the right panel corresponds to Y ~ Grade + Tim. The horizontal bars indicate the two Nim group means (after adjustment). In the first the two panels the bars are at the same height so Nim is not significant; on the right the bar at Nim = 1 is (significantly) higher than the bar at Nim = 0.

Now the interpretation is clearer: Tim has a stronger effect than either Grade or Nim. For example, Y is higher at Tim = 1 vs at Tim = 0, and Y is higher at Grade = 0 vs Grade 1. But the Grade effect is detected only if Tim is in the model: a strong effect explains a lot of the variability in Y, which reduces the residual error and makes it possible to detect smaller effects. Notice how the spread of the y-axis shrinks from left to right as we include more informative predictors.

To sum up, the type I ANOVA is not especially insightful; focus on understanding & reporting the type II AVOVA results. Plots always help with the interpretation. And finally, the design is not balanced: there are no measurements with Tim = 1 and Nim = 1. If feasible, you should "complete" the experiment; this will improve the analysis and make it possible to check for interactions between Nim and Tim.

• Thank you for the explanation, it make things clearer for me. Regardless of the lower effect of Grade and Nim i can still conclude based on the results that there is an effect, right? What would you report here? Since the group means are not appropriate what would you report? The adjusted group means? Or can i also report the coefficients of a linear model? Feb 15, 2023 at 9:23
• I don't find the summary of estimated coefficients easy to interpret. It can be more informative to report marginal effects & their comparisons instead. Take a look at the emmeans package and its vignettes. (Or search for emmeans on CV.) Also the ggeffects package with can make nice vizualizations of the marginal effects. Feb 15, 2023 at 10:00
• Okay, i will have a look at that, thank you. Just i dont understand why you find the summary of coefficients not easy to interpert. Does this not just say, if i go from 0 to 1 for a varaible the output changes by the corresponding coefficient? Feb 15, 2023 at 10:57
• This is my personal opinion of course. Take a look at the vignettes and the visualizations you can get with emmeans and ggeffects. Then compare them with the summary table. Try to interpret both and decide for yourself which one helps you describe the results of your analysis better. Feb 15, 2023 at 11:00
• I have an additional question. In your plot arent there triangles missing in the middle plot? Feb 15, 2023 at 18:45