# Different (Welch's) ANOVA tests and post hoc contradict paired tests. Which one to accept?

This is a problem that I think nothing would help but sharing my data here. I have 3 groups (n=7) which are paired. They are concentrations of a chemical compound in different parts fish tissue. For example, the 5th number in G1 and G2 and G3 are 3 different measurements from 3 different parts of the same fish. Thus, We examined 3 different parts of each of our 7 different fishes. Shapiro-Wilk and d'Agostino-Pearson tests are significant (p<0.05) but the Levene's test for means indicates the homogeneity of the data (p=0.57). Because of the outliers, I decided to run Welch's ANOVA instead of normal one-way ANOVA. The result is insignificant.

Some other researches on the same type of data, reported the log(10) values and ran ANOVA and observed significance between the groups of their own data. I did so, and the Log-transformation of my data is normal (Shapiro-Wilk Test) and homogeneous (Leven's test).

One way ANOVA (I'm not sure if it is a good idea to use normal ANOVA when n=7 only), is insignificant but when I use contrast post-hoc, there is a significance between G1 and G3. Welch ANOVA and Kruskal-Wallis tests are still insignificant when I use the Log values. Out of curiosity, I performed t-test for independent samples between all of my Log groups and there are significant differences between G1-G3, G2-G3 but G1-G2 is insignificant. Even the same t-tests on the raw data (before logarithms) shows exactly the same results. These results are in accordance with what other researchers have found with their dataset. Before each t-test, I ran a F-test to make sure my variances are not significant.

So, does it mean that I can accept the paired t-test although it is contrary to the omnibus Welch's ANOVA and other tests?

Update: I ran Mann-Whitney Test for Two Independent Samples since the normality of data was questionable and the results of the t-test were confirmed with non-parametric test too.

My Data are as follows:

G1     G2     G3
37.770  30.94  20.040
22.945  32.55  16.790
53.230  51.83  41.740
21.960  20.99  16.325
25.920  19.11  20.710
40.290  34.74  23.380
28.960  28.21  21.490

• What kind of data are these? – Frans Rodenburg Aug 21 '18 at 13:11
• Concentration of a chemical compound in different parts fish tissue. For example, the 5th number in G1 and G2 and G3 are 3 different measurements from 3 different parts of the same fish. We examined 7 fishes. – Hadi Aug 21 '18 at 14:57
• I see, well that is important information you should include in the question, because this means the data are dependent (from the same fish). You could model this with a mixed model. If you edit the question to include this I'll try to write out an answer. – Frans Rodenburg Aug 21 '18 at 15:04
• Thanks, I just revised it. So, don't we treat them like paired samples? – Hadi Aug 21 '18 at 15:53
• Paired data refers to grouping control/treatment observations into pairs of observations which are subtracted to reduce the influence of covariates. This could work if you only had two tissue types as the resulting numbers would be independent across fish, but with 3 types, this strategy falls apart. As Frans Rodenburg suggests, a mixed model with be a better choice. – deasmhumnha Aug 21 '18 at 17:38

In short: neither are appropriate and you shouldn't try to interpret them.

### Measurements from the same experimental units are not independent.

If G1, G2 and G3 come from the same 7 fish, then you cannot use a test or model that assumes independent observations. This is no different for non-parametric statistics.

### You can model this dependence using a linear mixed effects model.

By assuming that these fish come from a larger population of fish with random, normally distributed deviations from the 'average' fish w.r.t. the concentrations of these chemicals, we can estimate a random intercept for concentration.

In $\textsf{R}$, you can do this quite easily with the package lme4:

# A great package for mixed models
if(!require("lme4")){
install.packages("lme4")
library("lme4")
}

# A package for, among many other things, nice QQ-plots (including 95%-CI)
if(!require("car")){
install.packages("car")
library("car")
}

M <- data.frame(concentration = c(37.770, 30.94, 20.040,
22.945, 32.55, 16.790,
53.230, 51.83, 41.740,
21.960, 20.99, 16.325,
25.920, 19.11, 20.710,
40.290, 34.74, 23.380,
28.960, 28.21, 21.490),
bodypart      = factor(rep(1:3, times = 7)),
fish          = rep(1:7, each = 3))

# A Linear mixed effects model
LMM <- lmer(concentration ~ bodypart + (1 | fish), data = as.data.frame(M))

# Some basic diagnostics
plot(LMM)
qqPlot(residuals(LMM))

# A summary of the model
summary(LMM)

# Bootstrapped confidence intervals for difference from the reference group
confint(LMM, method = "boot")

# How to change the reference group:
M$bodypart <- relevel(M$bodypart, "2") # this would set G2 as reference group


### Why does this matter?

Your standard deviations (and thus any claims of significance) can differ quite a lot depending on how your model your data. In some cases you will underestimate your standard errors and in others you will overestimate them.

Take for example your own data and try and run a normal linear regression and a mixed model, using the same fixed effect (bodypart):

LM <- lm(concentration ~ bodypart, M)
summary(LM)
summary(LMM)


See the large difference in standard error? That's what happens when you treat measurements of the same fish as though they were independent.

This answer is already getting quite long and I recommend you read up on random and mixed effect models, but I'll try to give you an intuitive interpretation of why this difference can be so large, simply by plotting your data:

By calculating a random offset for each fish, we get a much clearer picture of what the effect of each body part is on the concentration.

Of course you could argue that a linear model with a fixed effect for fish also sees that these are different fish, however: (1) Body parts are still very much nested in fish, so this wouldn't solve the dependence issue (2) Estimating 7 different offsets would be a huge waste of your small sample size, when instead you can estimate a single random distribution from which your fish' offsets come.

I hope to have convinced you of the need to study mixed models a bit before trying out study designs with dependency.

• Good advice, however for this very simple design there is a well-known classical test, which is repeated measures ANOVA (RM-ANOVA). – amoeba says Reinstate Monica Aug 23 '18 at 13:47
• @frans-rodenburg I wish I knew more about the mixed model before I start. The experiments were designed 5 years ago and I am asked just lately to find the relationship between the concentrations and part of the fish. Since I have only 5 days to prepare my report, I think I won't have enough time to go deep in the concept of mixed models and how to use them for me data. I thank you for enlightening my mind about reconsidering the way I the independence of data and the useful information about mixed modelling. I don't know if RM-ANOVA is considered a mixed model. – Hadi Aug 24 '18 at 8:55
• As @amoeba mentioned, I thought I explored a bit about RM-ANOVA. I know it can be amusing to know that I'm using Excel for my tests thanks to the very wonderful resource and software I found at www.real-statistics.com. I believe my data match the descriptions here: real-statistics.com/anova-repeated-measures/… But seems my data reject the Sphericity and that means I cannot use RM-ANOVA. Do you think MANOVA would be the answer? – Hadi Aug 24 '18 at 8:58
• @Hadi, I have included a full example of how to run this model on your data in $\textsf{R}$, which is free software. If you look at e.g. the Wikipedia page of MANOVA you'll find that it is for modelling more than one response variable, it has nothing to do with dependence of observations. – Frans Rodenburg Aug 24 '18 at 9:08