My data is like the below table:

username       Tweet_id      Agegroup      Time     Score
A               unique        Young        1        0.2
A               unique        Young        2        0.3
B               unique        Old          1        0.5
c               unique        Old          4        0.6

Each user(username) may post between one and many tweets, and each tweet(tweet_id) was posted in different times(Time) and has a different score (Score).

I hope to examine the effect of time and age(also an interaction) on Score.

My guess is that tweetid should be nested in the username like a class nested in the school.

I am not sure if I am building up a correct linear mixed model. Since I have a million row, how I can speed up the test. Also, How can I generate linear mixed model tables and plot(better to have an interaction plot between age and Time)

lm1 = lmer(score~ 1+agegroup+Time+ (1|username/Tweet_id),data = new_data,control = lmerControl(calc.derivs = FALSE))
  • $\begingroup$ Please explain the data a bit more. Your model has id as a grouping variable but this is not in the data extract. Also if tweet_id is nested in username then tweet_id should not have the same level for all the usernames in your data extract. Also you say that score is the outcome, but in your model it says compound $\endgroup$ – Robert Long May 25 at 7:41
  • $\begingroup$ @RobertLong, Thanks, I edited the data. The outcome variable is the score. Should Tweet_id be nested in username? If so, is my R code correct? $\endgroup$ – wei May 25 at 7:44
  • $\begingroup$ I run this code on R, but it seems that it will take forever to run it, and I never get an error nor the result. $\endgroup$ – wei May 25 at 7:46
  • $\begingroup$ Please provide the exact eror that you get. Also, from the data, it doesn't look like tweet_idis nested within username because they are have value unique for users A, B and C $\endgroup$ – Robert Long May 25 at 7:59
  • $\begingroup$ @RobertLong, I don't have the error. Just not sure if I have the right model to run. It seems the correct model should be lmer(compound ~ agegroup + Time + (1 | username), data = sample)? $\endgroup$ – wei May 25 at 8:01

Your model :

score ~ 1 + agegroup + Time + (1|username/Tweet_id)

Does make sense, apart from you mentioned that you are interested in the interaction between agegroup and Time and that model does not fit an interaction.

I find it strange that all the Tweet_ids have exactly the same value: unique.

Setting that aside it seems your problem is that it takes a long time to fit. This is not surprising given you say you have around a million rows. A few things I can suggest:

  • Split that data into smaller chunks, say 25,000 rows and see how long the model takes to run 32 of those those. Then try 50,000 on 16 of those, then 100,000 on 8, then 200,000 on 4. From this you will be able to estimate how long it will take with the full dataset. FYI a few years ago I had some mixed models that took 15 days to run :o

  • check the resources that are being used by the machine you are running it on. If it is mainly CPU then run it on an instance with a faster CPU that is also optimised for compute.

  • $\begingroup$ @wei Does this answer your question ? If so please consider marking it as the accepted answer. If not, please let us know why. Also, if you haven't already, please consider upvoting it. $\endgroup$ – Robert Long Jun 26 at 12:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.