I would like to know if there is a significant effect of condition on Score1, and how closely correlated Score2 is at approximating Score1, for the following dataset:

This is the format of my dataset

I am using a random intercept with random slopes model in R: data1.model=lmer(Score1 ~ Condition+(1+Condition|Subject), data=data1,REML=FALSE)

Score2 is an indirect measure of Score1, in this type of experiment, Score2 is often the only available measurement and thus it is used as an approximation of Score1. However, in this experiment we have been able to measure both variables. So my question is, how effective is Score2 at approximating Score1? I think I am looking for the amount of variance in Score2 that can be explained by Score1.

I have looked at this problem using rmcorr: Bakdash, Jonathan Z., and Laura R. Marusich. "Repeated measures correlation." Frontiers in psychology 8 (2017): 456.

However, I think a mixed model may offer more flexibility as I have a third variable (Score3) influencing the Score2 measurement and I would like to find the amount of variance in Score2 that can be explained by Score1, while controlling for the third variable (which is also measured at each trial).

Any direction on this problem would be much appreciated!

  • $\begingroup$ How do you define “effective”? Do Score 1 & 2 measure the same thing, and you want to see agreement? Or you want to see if Score 2 accurately predicts Score 1? Or something else? $\endgroup$ – Dimitris Rizopoulos Sep 29 '18 at 21:30
  • $\begingroup$ Yes, Score1 and Score2 are measuring the same thing, Score1 is the most accurate way to measure this variable. With rmcorr, I found that Score1 & Score2 were significantly correlated with r=0.3 (p<.01), however, rmcorr does not allow me to include Score3 as a covariate. I would define Score2 as being an effective way to measure Score1, if Score1 accounts for approximately 30-40% in Score2 - when you remove any influence of changes in Score3. Though Score1 & Score2 are correlated, it is possible that Score2 does not accurately predict Score1 & is dominated by variability in Score3. $\endgroup$ – Cipriana Oct 1 '18 at 9:32
  • $\begingroup$ Are you sure you're interested in the correlation between Score 1 and Score 2, and not in agreement? $\endgroup$ – Dimitris Rizopoulos Oct 1 '18 at 10:43
  • $\begingroup$ I wasn't aware before that there were different statistical tests depending on whether you are looking at correlation and agreement - but I think as I am measuring the same variable I am interested in the agreement. $\endgroup$ – Cipriana Oct 1 '18 at 15:52

Based on the comments above, it seems that you are more interested in the agreement between Score1 and Score2. Note that correlation and agreement are not the same things. For example, if Score1 and Score2 perfectly agree, and you create a new variable Score2*, which is Score2 + 10, then Score2* and Score1 do not agree anymore. However, the correlation between Score1 and Score2 equals the correlation between Score1 and Score2*.

If you're interested in the agreement between two continuous variables, you could use the Bland-and-Altman Plot that lets you investigate how well the two Scores agree, and whether the differences between them are affected by the level (average) of the score (e.g., that the agreement is better for low scores than for high scores). You could also do it differently per condition.

The fact that you have repeated measurements means that you will need to calculate the limits of agreements based on the results of a mixed models analysis of the differences between the two scores in which you include as a fixed effect the average of the two scores.

  • $\begingroup$ Your analogy is very useful - Score2 is not measured on the same scale as Score1 and is likely to be offset from Score1 - so maybe it is correlation and not agreement I am looking for. My hypothesis is that changes in Score2 can be used to predict changes in Score1, but the agreement of their baseline values does not matter. The second thing I need to test is that changes in Score2 can still be used to predict changes in Score1, when Score3 is introduced as a covariate (as this will influence the Score2 measurement). $\endgroup$ – Cipriana Oct 3 '18 at 9:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.