# Ancova: How to proceed when residuals are not normal distrubuted but transformation changes meaning of results

I am trying to analyze water nutrient (DIN) at three different sites along distance transects perpendicular to the coast. The transects were sampled in two years (year t1 and t2) in two seasons (1 and 2).

I am trying to do a linear regression over the distance. As the residuals were not normal distributed I tried a boxcox transformation (0.26).

After model validation I did a post-hoc test using emmeans and tukey HSD. The results of post-hoc-test on the transformed data show different results than on the not-transformed data. Ignoring the effect of distance for now and using the boxcox transformation (Transformed.jpg), at t1 season 2 is significantly lower than all other groups. When using no transformation, at t1 season 2 is still significantly lower but at t2 season 2 is significantly higher than all others.

Looking at the not-transformed data (Untransformed.jpg) it looks like as if t2 season 2 is higher than the others. I can think of an reasonable explanation why DIN is lower in season 2 in year t1 but higher in the same season in year t2.

I am not sure what to do since either the assumptions of anova are not met (normality of residuals) or the results are diverted by the transformation. What should I do? What are the alternatives to transforming the data?

Here's my script:

mydata<-read.table("din.test.txt",header=T)

require(MASS)
require(car)
#### model validation ####
boxcox(din~site*season*dist*year, data=mydata,lambda = seq(0.25,0.28,length=50))

m1<-lm(din^.26~site*season*dist*year,data=mydata)
Anova(m1)
plot(m1)

# model reduction according to Anova results in this model:
m9<-lm(din^.26~ season*year +  site*year + dist + year + season + site,data=mydata)
Anova(m9)
plot(m9)
#model without boxcox transformation:
m9x<-lm(din~ season*year +  site*year + dist + year + season +site,data=mydata)

# looking at the season*year interaction
require(emmeans)
cld(a)

cld(b)
# plotting the season*year interaction
require(sciplot)
sciplot::bargraph.CI(year,din,group=season,legend=T,data=mydata)
sciplot::bargraph.CI(year,din^.26,group=season,legend=T,data=mydata)


And here's the data (sorry, I can't add a file):

"site"  "season"    "dist"  "year"  "din"
"YL"    "2" 0.25    "t1"    0.331
"YL"    "2" 0.5 "t1"    0.982
"YL"    "2" 0.1 "t1"    0.118
"YL"    "2" 0.05    "t1"    0.129
"QG"    "2" 0.05    "t1"    2.499
"QG"    "2" 0.1 "t1"    0.189
"QG"    "2" 0.25    "t1"    0.162
"QG"    "2" 0.5 "t1"    0.093
"QG"    "2" 1   "t1"    0.142
"YL"    "2" 0.1 "t1"    0.36
"YL"    "2" 0.25    "t1"    0.478
"YL"    "2" 0.05    "t1"    1.038
"YL"    "2" 0.5 "t1"    1.1
"CQ"    "2" 1   "t1"    5.621
"QG"    "2" 0.05    "t1"    21.71
"QG"    "2" 0.25    "t1"    1.679
"QG"    "2" 0.1 "t1"    4.08
"QG"    "2" 1   "t1"    1.94
"QG"    "2" 0.5 "t1"    2.311
"CQ"    "2" 1   "t1"    14.099
"CQ"    "2" 0.5 "t1"    14.574
"CQ"    "2" 0.25    "t1"    22.676
"CQ"    "2" 0.05    "t1"    17.463
"CQ"    "2" 0.1 "t1"    20.244
"YL"    "2" 0.1 "t1"    0.044
"YL"    "2" 0.25    "t1"    2.044
"YL"    "2" 0.05    "t1"    0.869
"YL"    "2" 0.5 "t1"    1.11
"QG"    "2" 0.25    "t1"    0.63
"QG"    "2" 0.05    "t1"    2.35
"QG"    "2" 1   "t1"    0.525
"QG"    "2" 0.1 "t1"    0.831
"QG"    "2" 0.5 "t1"    0.016
"CQ"    "2" 1   "t1"    0.029
"CQ"    "2" 0.5 "t1"    0.113
"CQ"    "2" 0.25    "t1"    0.008
"CQ"    "2" 0.1 "t1"    0.125
"CQ"    "2" 0.05    "t1"    0.117
"YL"    "1" 0.25    "t1"    0.794551941
"YL"    "1" 0.1 "t1"    0.802065535
"YL"    "1" 0.5 "t1"    1.053975465
"QG"    "1" 0.5 "t1"    1.499067821
"QG"    "1" 0.25    "t1"    1.252671444
"QG"    "1" 0.05    "t1"    7.946370823
"QG"    "1" 0.1 "t1"    2.775810782
"QG"    "1" 1   "t1"    2.437018455
"QG"    "1" 0.25    "t1"    3.902968159
"QG"    "1" 0.5 "t1"    3.246770597
"QG"    "1" 0.05    "t1"    9.450082588
"QG"    "1" 0.1 "t1"    13.47107368
"QG"    "1" 1   "t1"    7.492079603
"YL"    "1" 0.25    "t1"    1.900188587
"YL"    "1" 0.05    "t1"    2.021900961
"YL"    "1" 0.1 "t1"    2.278321923
"YL"    "1" 0.5 "t1"    1.871895111
"QG"    "1" 0.25    "t1"    2.294669047
"QG"    "1" 0.5 "t1"    1.888579214
"QG"    "1" 0.05    "t1"    10.04203514
"QG"    "1" 0.1 "t1"    3.986006555
"YL"    "1" 0.1 "t1"    3.87514935
"YL"    "1" 0.25    "t1"    6.39091688
"YL"    "1" 0.05    "t1"    5.823291513
"YL"    "1" 0.5 "t1"    5.090676138
"CQ"    "1" 0.25    "t1"    16.23256302
"CQ"    "1" 1   "t1"    11.62109699
"YL"    "1" 0.1 "t1"    5.555394629
"YL"    "1" 0.25    "t1"    5.431119976
"CQ"    "1" 0.5 "t1"    11.10402276
"CQ"    "1" 0.1 "t1"    21.72465973
"YL"    "1" 0.05    "t1"    5.915380909
"CQ"    "1" 0.05    "t1"    24.48758476
"QG"    "1" 0.25    "t1"    2.85583221
"CQ"    "1" 0.25    "t1"    13.30408629
"CQ"    "1" 1   "t1"    6.376336306
"CQ"    "1" 0.5 "t1"    8.941315787
"CQ"    "1" 0.1 "t1"    22.38464481
"QG"    "1" 0.5 "t1"    2.68250619
"QG"    "1" 0.05    "t1"    6.916896531
"QG"    "1" 1   "t1"    3.774857706
"QG"    "1" 0.1 "t1"    2.391395439
"CQ"    "1" 0.05    "t1"    20.42033651
"QG"    "1" 0.25    "t1"    2.8652283
"QG"    "1" 0.5 "t1"    2.250031469
"QG"    "1" 0.05    "t1"    7.630071648
"QG"    "1" 1   "t1"    2.886982774
"QG"    "1" 0.1 "t1"    3.827006474
"YL"    "1" 0.25    "t1"    7.794439577
"YL"    "1" 0.05    "t1"    6.248307783
"YL"    "1" 0.1 "t1"    6.528010444
"YL"    "1" 0.5 "t1"    6.200407013
"QG"    "1" 0.25    "t1"    2.889264534
"QG"    "1" 0.05    "t1"    7.890414858
"QG"    "1" 0.1 "t1"    5.148877549
"QG"    "1" 0.5 "t1"    2.029281633
"QG"    "1" 0.25    "t1"    3.904111773
"QG"    "1" 0.1 "t1"    6.246946805
"QG"    "1" 0.05    "t1"    6.360074604
"QG"    "1" 0.5 "t1"    5.626000429
"YL"    "1" 0.25    "t1"    8.014156906
"QG"    "1" 1   "t1"    3.805575929
"YL"    "1" 0.05    "t1"    8.343367495
"QG"    "1" 0.05    "t1"    17.34552216
"YL"    "1" 0.1 "t1"    7.882847092
"QG"    "1" 0.1 "t1"    3.97992341
"QG"    "1" 0.25    "t1"    3.397216912
"YL"    "1" 0.5 "t1"    7.508480929
"QG"    "1" 0.5 "t1"    2.538726743
"CQ"    "1" 1   "t1"    2.609908718
"CQ"    "1" 0.1 "t1"    11.96074271
"QG"    "1" 0.1 "t1"    8.356295507
"CQ"    "1" 0.25    "t1"    1.891502384
"CQ"    "1" 0.5 "t1"    2.411142987
"QG"    "1" 0.25    "t1"    3.076694944
"QG"    "1" 1   "t1"    2.425160675
"QG"    "1" 0.05    "t1"    10.58729615
"CQ"    "1" 0.05    "t1"    14.21987312
"QG"    "1" 0.5 "t1"    3.051606935
"YL"    "1" 0.25    "t1"    2.91024084
"YL"    "1" 0.5 "t1"    3.92622086
"YL"    "1" 0.05    "t1"    3.350444753
"YL"    "1" 0.1 "t1"    2.835182398
"YL"    "1" 0.1 "t1"    8.56766154
"YL"    "1" 0.5 "t1"    5.999452523
"YL"    "1" 0.05    "t1"    7.984721546
"YL"    "1" 0.25    "t1"    11.60931087
"YL"    "1" 1   "t1"    2.86563723
"QG"    "1" 0.25    "t1"    5.330971216
"QG"    "1" 0.1 "t1"    7.301651503
"QG"    "1" 1   "t1"    8.458483787
"QG"    "1" 0.05    "t1"    14.9215408
"QG"    "1" 0.5 "t1"    6.858708763
"CQ"    "1" 0.1 "t1"    9.076500393
"CQ"    "1" 1   "t1"    5.203014916
"CQ"    "1" 0.5 "t1"    6.884880258
"CQ"    "1" 0.25    "t1"    8.613154696
"CQ"    "1" 0.05    "t1"    5.795466633
"YL"    "1" 0.1 "t1"    10.79281488
"YL"    "1" 0.5 "t1"    41.14463184
"YL"    "1" 0.05    "t1"    14.85242575
"YL"    "1" 0.25    "t1"    21.92891246
"YL"    "1" 1   "t1"    3.198003695
"YL"    "1" 0.1 "t1"    2.793367655
"YL"    "1" 0.5 "t1"    7.113345434
"YL"    "1" 0.05    "t1"    4.080651172
"YL"    "1" 0.25    "t1"    5.866714137
"YL"    "1" 1   "t1"    3.405779353
"QG"    "1" 0.25    "t1"    2.043567782
"QG"    "1" 0.1 "t1"    1.157633687
"QG"    "1" 1   "t1"    2.166830815
"QG"    "1" 0.05    "t1"    1.312962779
"QG"    "1" 0.5 "t1"    4.874829305
"YL"    "1" 0.1 "t1"    1.255747112
"YL"    "1" 0.05    "t1"    1.844841825
"YL"    "1" 0.5 "t1"    1.751416066
"YL"    "1" 0.25    "t1"    2.62786284
"YL"    "1" 0.1 "t2"    3.795
"YL"    "1" 0.05    "t2"    1.324
"YL"    "1" 0.25    "t2"    1.393
"YL"    "1" 0.25    "t2"    1.271
"YL"    "1" 0.05    "t2"    1.853
"YL"    "1" 0.25    "t2"    2.139
"YL"    "1" 0.25    "t2"    1.873
"YL"    "1" 0.25    "t2"    1.21
"YL"    "1" 0.25    "t2"    2.744
"QG"    "1" 0.25    "t2"    2.073
"QG"    "1" 0.05    "t2"    36.337
"QG"    "1" 0.1 "t2"    1.809
"QG"    "1" 0.25    "t2"    1.8
"QG"    "1" 0.5 "t2"    2.52
"YL"    "1" 0.5 "t2"    2.025
"YL"    "1" 0.05    "t2"    1.509
"YL"    "1" 0.25    "t2"    1.067
"YL"    "1" 0.25    "t2"    1.859
"YL"    "1" 0.5 "t2"    1.535
"YL"    "1" 0.1 "t2"    1.44
"YL"    "1" 0.05    "t2"    1.751
"QG"    "1" 0.25    "t2"    2.108
"QG"    "1" 0.05    "t2"    2.725
"QG"    "1" 0.1 "t2"    21.652
"QG"    "1" 0.5 "t2"    1.349
"QG"    "1" 1   "t2"    1.688
"CQ"    "1" 0.5 "t2"    16.46
"CQ"    "1" 1   "t2"    13.465
"CQ"    "1" 0.25    "t2"    16.641
"CQ"    "1" 0.1 "t2"    8.975
"CQ"    "1" 0.05    "t2"    18.141
"YL"    "1" 0.1 "t2"    1.316
"CQ"    "2" 0.25    "t2"    24.7
"CQ"    "2" 1   "t2"    15.25
"CQ"    "2" 0.5 "t2"    10.53
"CQ"    "2" 0.05    "t2"    11.11
"CQ"    "2" 0.1 "t2"    44.23
"CQ"    "2" 0.1 "t2"    26.57
"CQ"    "2" 1   "t2"    12.88
"CQ"    "2" 0.05    "t2"    49.92
"CQ"    "2" 0.5 "t2"    22.02
"CQ"    "2" 0.25    "t2"    18.66
"QG"    "2" 0.5 "t2"    6.05
"QG"    "2" 1   "t2"    6.15
"QG"    "2" 0.25    "t2"    13.62
"QG"    "2" 0.05    "t2"    14.18
"QG"    "2" 0.1 "t2"    19.58
"QG"    "2" 0.1 "t2"    22.79
"QG"    "2" 0.05    "t2"    26.69
"QG"    "2" 0.5 "t2"    7.88
"QG"    "2" 1   "t2"    5.96
"QG"    "2" 0.25    "t2"    15.21
"YL"    "2" 0.25    "t2"    0.01
"YL"    "2" 0.5 "t2"    1.82
"YL"    "2" 0.1 "t2"    0.18
"YL"    "2" 0.1 "t2"    0.29
"YL"    "2" 0.05    "t2"    0.11
"YL"    "2" 0.25    "t2"    0.26
"YL"    "2" 0.5 "t2"    0.92
"YL"    "2" 0.05    "t2"    0.04
"CQ"    "2" 0.05    "t2"    31.58
"YL"    "2" 0.25    "t2"    7.45
"QG"    "2" 0.25    "t2"    0.61
"CQ"    "2" 1   "t2"    11.16
"QG"    "2" 0.05    "t2"    20.91
"QG"    "2" 0.1 "t2"    0.66
"YL"    "2" 0.05    "t2"    12.2
"YL"    "2" 0.1 "t2"    12.05
"YL"    "2" 0.25    "t2"    13.12