My experiment consist on measuring the root mean squared error (RMSE) between a path and what the participant produces. To produce the trace each participant experiences 3 feedbacks (1st factor - categorical), and each feedback has a bandwidth of either 2 or 5 Hz (2nd factor - numerical). Finally, each trace can have a mean value of 10 or 15 (3rd factor - numerical). Each participant experiences all the possible combinations (3 feedbacks * 2 frequencies * 2 mean levels = 12 combinations) twice. Hence, there are 24 RMSE measurements per participants. I have 7 participants.
The model I am fitting is
model_error <-lme(error ~ feedback * frequency * force_level * motor_units, random = ~ 1 | subject, data=df)
I am using the
nlme library in R to implement it and
emmeans to test for differences between the marginal means.
When checking for the soundness of the model, as expected from a dependent variable with a lower bound (i.e. non negative), the residual plot presents heteroscedasticity (see image below).
After using the log transform for the
error and fitting the model again, the situation normalises.
What bothers me is that when looking for significant differences, such as
emmeans(model_error, pairwise ~ feedback)
some results change drastically, going from p-values of 0.0001 to 0.3.
Can someone please help me on understanding whether the log transform is necessary and how to proceed in any case?
I also tried the Box-Cox transform with lambda = -0.6 (the value was given as the optimal lambda by the
boxcoxmix package). The QQ plot looks even nicer with respect to the log transform, but still: which results from
emmeans should I believe? Before or after the transformation? Why?
UPDATE 2 These are the residuals after Box-Cox transformation