I want to evaluate how well a device (dev.B) can predict accurately the values of another device (dev.A) that is used as a reference device. dev.B gives a value every specific time intervals (e.g. 120 seconds), and what I do is to make an averaged value per hour for the dev.B. What I want is to know if the ability of dev.B to predict values from the dev.A depends on the number of values I have per hour (some times I might have two values in one hour and others I might have 30). For the dev.A I have a unique value representative of the whole hour. Important: both devices were placed simultaneously in different animals. A fake example of the original data frame I have is below:

DateTime                 n.data      dev.A       dev.B     ID

2013-08-14 12:00:00         8        1.453       1.153      A
2013-08-14 13:00:00         3        0.653       0.834      A
2013-08-14 14:00:00        21        2.953       2.098      A
          .                 .          .           .        .
          .                 .          .           .        .
          .                 .          .           .        .

Since my purpose is to show the importance of having a large number of data from the device dev.B per hour to predict values accurately from the device dev.A, what I did is to create a categorical variable called data-quantity which classified n.data as low, medium or high, and then I run an LMM for each data-quantity. I used ID as random effect since I am interested in knowing the importance of the data-quantity in general, without considering the individual. In future studies, researchers will have only data from the dev.B for the different individuals, and thus, it is nos possible to calculate different intercepts and slopes for the different IDs. On the other hand, I think that is easier for future readers to understand the purposes of my study if I compare the explained variance of dev.A using dev.B when I have low, medium and high data-quantities instead of running one model which uses as variable of fixed factor n.data. Below I show the original data frame with the variable data-quantity:

DateTime                 n.data   data.quantity  dev.A       dev.B     ID

2013-08-14 12:00:00         8         Medium     1.453       1.153      A
2013-08-14 13:00:00         3         Low        0.653       0.834      A
2013-08-14 14:00:00        21         High       2.953       2.098      A
          .                 .           .          .           .        .
          .                 .           .          .           .        .
          .                 .           .          .           .        .

The code for the LMMs were:

mod1<-lmer(dev.A ~ dev.B + (dev.B|ID),data =data[data$data.quantity=="High",],REML=F)
mod2<-lmer(dev.A ~ dev.B + (dev.B|ID),data =data[data$data.quantity=="Medium",],REML=F)
mod3<-lmer(dev.A ~ dev.B + (dev.B|ID),data =data[data$data.quantity=="Low",],REML=F)

Relative to the explained variance, I got this:

Model   r2m      r2c

 m1    0.5518   0.9038
 m2    0.5510   0.8460
 m3    0.5168   0.6864

My doubt is about how to interpret that. Looking at r2m seems that the quantity of data per hour for the device dev.B is not critical since with low data quantity the decrease in variance explained compared to when the data quantity is high is only about 4%. However, if you look at r2c, you can see a sharper difference among data-quantities for dev.B in their ability to predict values for dev.A.

How should I interpret those results in terms of my purpose? Can I say with those results that data-quantity is important for predicting values of dev.A?

Thanks in advance.

  • 1
    $\begingroup$ Why have you split the data ? It may be better to fit the model on the whole dataset: lmer(dev.A ~ dev.B*data.quantity + (dev.B|ID),data =data) then the main effects and interactions will directly answer your research question. $\endgroup$ Aug 6 '20 at 12:05
  • $\begingroup$ To simplify I didn't indicate that in my real case I use different time windows (not only 1 hour, but also 15 minutes, 30 minutes and 2 hours). So I have dev.B, n.data (or data.quantity depending on what I use) and Time.window. Then, if I include everithyng in a unique model I should include the interaction between those three fixed variables and I think that it is easier to understand if I split in three levels of data. However, I will deep on your recommendation. But still, I would like to know in this scenario how should I interpret the results... $\endgroup$
    – Dekike
    Aug 6 '20 at 12:43
  • $\begingroup$ My advice is not to use r2m or r2c and fit a model on your full dataset. See here $\endgroup$ Aug 6 '20 at 12:45
  • $\begingroup$ Thanks! If I do it as you say, I was thinking how to figure out the importance of the factor n.data when predicting dev.A. How can I quantify that? I guess the coefficient for this fixed factor gives an idea, but I am also thinking if I can calculate r2m with and without n.data' and the difference is telling the importance of n.data`. Does it make sense? $\endgroup$
    – Dekike
    Aug 6 '20 at 12:52
  • $\begingroup$ What do you mean by "the importance of the factor" ? $\endgroup$ Aug 6 '20 at 13:09

Aside from the comments about running this as one model (which I agree with)--I also think you might consider reporting a different outcome altogether.

If your goal is to show prediction, then I think it would be most relevant to report the accuracy of model predictions. In other words you could report the testing/training error for each model. Then your outcome is direct a measure of what you are trying to show. Its not that r2 is wrong here, its just one step removed from the literal prediction error which it sounds like you are more interested in.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.