I am analyzing data using Liner Mixed Effects Model (LMM) with lme4 package in R. After visualizing the prediction slope and observation on a scatter plot, I realized that the estimation for the one of factor (e.g., treatment 3 in the illustration) is far from observation. after checking the data, I found that one of the nested variables (i.e. random effects) has only one observation, so that including the random intercept is adjusting the slope for the factor too much.

The following is the illustration of the very simplified research design.

If I include participant_id as a random effect, the estimation for treatment 3 does not consider the participant C's observatio at all, which is probably cancelled out by random intercept for participant C.

res <- lmer (value ~ treatment_type + (1|participant_id))

Illustration of the data structure

I am aware if I don't use the mixed effect model, I am vaiolating the independent observation assumption. Because of the pusrpose of research, I really cannot just exlude treatment 3.

So, the approach I am thinking is: model excluding random intercept of participant_it. Then, compare both models for the estimated values for coefficients and their CIs. If they are not drastically different, I would choose the res2, the model without the random effect.

require(LmerTest)   ##in order to attain p-values
res2 <- lm (value ~ treatment_type)
  1. Is my approach appropriate one?

  2. If my approach is appropriate; when including and excluding random effects generates significant differences among estimations, what should I do?

  3. What are the other options I can use for this kind of data set?

Thank you for your help!

Detailed information about my data structure

The actual structure of my data is bit more complicated than illustrated in the image. I have data from about 40 participants nested within 15 groups (in which participants learned to-be-learned items together). I am examining the 5 types of learning conditions. Two or three of the groups engaged in more than one type of learning condition. Some participants engaged in only one types of learning condition by themselves. Some groups were also measured by the effect of learning repeatedly (i.e., immediate posttests and delayed posttests). Actually, less than half of the group only took delayed posttest. Some of the groups took more than one types of measurements, whereas the other groups just took one tyeps of measurement.

My ideal model

The following is my ideal model:

    ideal_model <- lmer (value ~ learning_condition * measurement_type * test_timing +
(1| measurement_type:participant_id)+

I am sorry if explanation is confusing. I can elaborate the explanation more in case I need to.

  • $\begingroup$ Based on your image does your data include 4 subjects with a total of 6 observations or is that just illustrative? I ask because for a mixed effects design, that is going to be too few subjects and too few observations. Specifying a random effect may be the least of your concerns if hypothesis testing is a goal. $\endgroup$ – Matt Barstead Jan 9 '18 at 0:09
  • $\begingroup$ Thank you for your question, @MattBarstead. The image is just illustrative; however, there are some clustered like Participant C or Participant D, where only one observation for each treatment was attained. Hypothesis testing is the second thing I would like to do. The main purpose is to attain confidence intervals for each treatment. If you would not mind, could you explain more about 'Specifying a random effect may be the least of your concerns if hypothesis testing is a goal'? Thank you. $\endgroup$ – user8460166 Jan 9 '18 at 2:53
  • $\begingroup$ The actual research design is a bit more complicated. I am including three factors: treatment type, test type, and test timing. Some of the participants experienced more than one treatment and took more than one measurement, multiple times. However, it is not the case for all participants. Although I do have a fairly abundant number of observations, because I am examining 3 way interactions of the abovementioned three factors, some of the participants are not fully nested, e.g., some participants like C & D in the image, took only one treatment. I hope this explanation makes sense. $\endgroup$ – user8460166 Jan 9 '18 at 3:01
  • $\begingroup$ My concerns were mostly about power and, if there were so few observations, obtaining accurate estimates of the variance for the error terms and the random effects overall. Having some cases with only one observation is not necessarily problematic. The other individuals in the model with multiple observations should effectively compensate when it comes to estimating the variance of the random effects (and the level 2 residuals). If you could provide a more comprehensive and reproducible example I may be able to offer some additional guidance (i.e., structure of the data & desired model). $\endgroup$ – Matt Barstead Jan 9 '18 at 16:13
  • $\begingroup$ I see. If I understand accurately, if I have a data set which is large enough, I would not have to care about some of the cases with only one observations; however if not, I have to be cautious. By "The other individuals in the model with multiple observations should effectively compensate when it comes to estimating the variance of the random effects (and the level 2 residuals).", you mean that that would not be problem to include random slope for interaction between participant_id and 'test_timing` (i.e., +(1|test_timing:participant_id)). Am I understanding correctly? $\endgroup$ – user8460166 Jan 10 '18 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.