0
$\begingroup$
  • Situation:

I'm analyzing results from an experiment with a hierarchical structure. There are two groups A and B, multiple subjects within each group, and multiple measurements within each subject. The treatment was applied to subjects, so individual measurements are just replicates of the same condition within each subject. No subject got both treatments A and B. I'm interested in finding out, whether the main effect (A vs B) is significant. I've attempted to achieve this using a mixed linear model with measurement being the dependent variable, A vs B being a fixed effect, and subject being a random effect.

  • Outcome/issue:

It looks like the random effect subject is accounting for variance that should belong to the main effect treatment instead. It's "syphoning away" variance from the treatment. Accordingly, my main effect appears to have barely any effect. What made me come to this conclusion? When I run a simple t-test with marginal means (took averages from each subject), my main effect comes out significant.

  • Questions:

Am I doing something fundamentally wrong here? If not, is there something I can do to overcome this issue? I was thinking if there was a way to force the random factor to only account for variance unexplained by the fixed effect.

I highly appreciate help in any format.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ I don't think you're doing anything fundamentally wrong. However, by taking the mean of all measurements for an individual, you lose all the variability that was associated to individuals. If the variability associated to individuals is very high, it might affect 'how significant' your effect is. However it is not clear what you mean by 'my main effect seems to have barely any effect.' Do you mean that it is not significant anymore? Or that the effect is now small (as in the coefficient is different and now the effect size is small)? $\endgroup$
    – Tilt
    Jan 18, 2021 at 19:58
  • $\begingroup$ Many thanks for getting back to me! Simply taking the mean is exactly what I intend to avoid. I want to consider those within-individual datapoints. That's how I ended up experimenting with these mixed effects models. Yes, by saying "my main effect seems to have barely any effect" I meant it was not significant anymore. The reason I mentioned running a t-test on marginal means and getting significant results was to demonstrate my suspicion that there probably is a significant effect, and I can't explain why the mixed effects model is suggesting otherwise. $\endgroup$
    – Haliaetus
    Jan 18, 2021 at 20:37

1 Answer 1

Reset to default
1
$\begingroup$

There's no way I know of to give priority to a certain effect rather than another to partition the variance (other than coding your model by hand yourself), and that is likely for the reason that it's exactly what you don't want to be doing in general. The reason you're using random effects of 'subjects' is to make sure that there is indeed a significant difference between A and B, and not a difference due to sampling the same individuals multiple times. What could be going on is also that the difference between A and B is in fact not that clear. The algorithms will partition the variance trying to find the best possible solution to your maximum likelihood equation (or other approximations of ML depending on the model/function/program you're using). I suggest you take a look at this great thread on random effects: What is the difference between fixed effect, random effect and mixed effect models?, which gives good examples of how and why they are important to consider. The answer by Christoph gives a great example where you might think that Y is strongly positively affected by X, but realize that this is simply due to individual(stacks) differences.

Good luck!

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you for your help, again! I'm digging through the resources you suggested. I do want to point out that my main X variable is specific to subject, not individual measurements. So in my case, both the fixed effect and random effect are tied to subject level. This differs from Christoph's answer. $\endgroup$
    – Haliaetus
    Jan 18, 2021 at 23:08
  • 1
    $\begingroup$ You're right, it is not quite the same but I was trying to find an example of how an apparent relationship does not hold once some other variable is taken into account. Do you get the same kind of result if you include subject as a fixed effect? (not that it should be a fixed effect, but it can at least confirm that all is right with your analysis if the result for the effect of your treatment is similar in both analyses) $\endgroup$
    – Tilt
    Jan 18, 2021 at 23:17
  • $\begingroup$ I have not tried that yet. Before I do so though, I'd like to better understand the fixed effect approach. According to my current level of understanding, fixed effects are not constrained into a distribution, meaning that each subject has its own dummy variable as fixed effect, and this fixed effect is free to take up the whole deviation from the sample mean, meaning that there won't be any variance left for the actual main factor. Is this correct? $\endgroup$
    – Haliaetus
    Jan 18, 2021 at 23:57
  • 1
    $\begingroup$ I'm no statistician so I don't want to induce errors or say on things I'm not sure about. What I know is that both random or fixed effects are free to 'take up the whole deviation'. Neither is constrained in the amount of variation it can explain in your data. To obtain the MLE, the algorithm is trying out a plethora of combinations of parameter values for your different effects until the combination that fits your data best is reached. There's no prioritization of 'main' or'secondary' effects.. I mean, how did you decide that your RE is accounting for variance 'belonging' to the other effect? $\endgroup$
    – Tilt
    Jan 19, 2021 at 13:33
  • $\begingroup$ Really appreciate your insights, I found them very useful. I now have a better understanding of what's going on. $\endgroup$
    – Haliaetus
    Jan 22, 2021 at 3:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.