Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I just came across this paper, which describes how to compute the repeatability (a.k.a. reliability, a.k.a. intraclass correlation) of a measurement via mixed effects modelling. The R code would be:

#fit the model
fit = lmer(dv~(1|unit),data=my_data)

#obtain the variance estimates
vc = VarCorr(fit)
residual_var = attr(vc,'sc')^2
intercept_var = attr(vc$id,'stddev')[1]^2

#compute the unadjusted repeatability
R = intercept_var/(intercept_var+residual_var)

#compute n0, the repeatability adjustment
n = as.data.frame(table(my_data$unit))
    k = nrow(n)
    N = sum(n$Freq)
n0 = (N-(sum(n$Freq^2)/N))/(k-1)

#compute the adjusted repeatability
Rn = R/(R+(1-R)/n0)

I believe that this approach can also be used to compute the reliability of effects (i.e. sum contrast effect of a variable with 2 levels), as in:

#make sure the effect variable has sum contrasts
contrasts(my_data$iv) = contr.sum

#fit the model
fit = lmer(dv~(iv|unit)+iv,data=my_data)

#obtain the variance estimates
vc = VarCorr(fit)
residual_var = attr(vc,'sc')^2
effect_var = attr(vc$id,'stddev')[2]^2

#compute the unadjusted repeatability
R = effect_var/(effect_var+residual_var)

#compute n0, the repeatability adjustment
n = as.data.frame(table(my_data$unit,my_data$iv))
k = nrow(n)
N = sum(n$Freq)
    n0 = (N-(sum(n$Freq^2)/N))/(k-1)

#compute the adjusted repeatability
Rn = R/(R+(1-R)/n0)

Three questions:

  1. Do the above computations for obtaining the point estimate of the repeatability of an effect make sense?
  2. When I have multiple variables whose repeatability I want to estimate, adding them all to the same fit (e.g. lmer(dv~(iv1+iv2|unit)+iv1+iv2) seems to yield higher repeatability estimates than creating a separate model for each effect. This makes sense computationally to me, as inclusion of multiple effects will tend to decrease the residual variance, but I'm not positive that the resulting repeatability estimates are valid. Are they?
  3. The above cited paper suggests that likelihood profiling might help me obtain confidence intervals for the repeatability estimates, but so far as I can tell, confint(profile(fit)) only provides intervals for the intercept and effect variances, whereas I would additionally need the interval for the residual variance to compute the interval for the repeatability, no?
share|improve this question

1 Answer 1

I think I can answer your questions at least concerning the unadjusted repeatability estimates, i.e., the classical intra-class correlations (ICCs). As for the "adjusted" repeatability estimates, I skimmed over the paper you linked and didn't really see where the formula that you apply can be found in the paper? It is also not immediately obvious to me what the adjustment is doing, so I will have to ignore the adjusted part of your question. It's not clear that this is a critical part of your question anyway. But if you think that it is, perhaps you can provide some clarification about the adjustment.

(1.) Do the above computations for obtaining the point estimate of the repeatability of an effect make sense?

Yes, the expression you propose does make sense, but a slight modification to your proposed formula is necessary. Below I show how one could derive your proposed repeatability coefficient. I hope this both clarifies the conceptual meaning of the coefficient and also shows why it would be desirable to modify it slightly.

To start off, let's first take the repeatability coefficient in your first case and clarify what it means and where it comes from. Understanding this will help us to understand the more complicated second case.

Random intercepts only

In this case, the mixed model for the $i$th response in the $j$th group is $$ y_{ij} = \beta_0 + u_{0j} + e_{ij}, $$ where the random intercepts $u_{0j}$ have variance $\sigma^2_{u_0}$ and the residuals $e_{ij}$ have variance $\sigma^2_e$.

Now, the correlation between two random variables $x$ and $y$ is defined as $$ corr = \frac{cov(x, y)}{\sqrt{var(x)var(y)}}. $$

The expression for ICC / repeatability coefficient then comes from letting the two random variables $x$ and $y$ be two observations drawn from the same $j$ group, $$ ICC = \frac{cov(\beta_0 + u_{0j} + e_{i_1j}, \beta_0 + u_{0j} + e_{i_2j})}{\sqrt{var(\beta_0 + u_{0j} + e_{i_1j})var(\beta_0 + u_{0j} + e_{i_2j})}}, $$ and if you simplify this using the definitions given above and the properties of variances/covariances (a process which I will not show here, unless you or others would prefer that I did), you end up with $$ ICC = \frac{\sigma^2_{u_0}}{\sigma^2_{u_0} + \sigma^2_e}. $$ What this means is that the ICC or "unadjusted repeatability coefficient" in this case has a simple interpretation as the expected correlation between a pair observations from the same cluster (net of the fixed effects, which in this case is just the grand mean). The fact that the ICC is also interpretable as a proportion of variance in this case is coincidental; that interpretation is not true in general for more complicated ICCs. The interpretation as some sort of correlation is what is primary.

Random intercepts and random slopes

Now for the second case, we have to first clarify what precisely is meant by "the reliability of effects (i.e. sum contrast effect of a variable with 2 levels)" -- your words.

First we lay out the model. The mixed model for the $i$th response in the $j$th group under the $k$th level of a contrast-coded predictor $x$ is $$ y_{ijk} = \beta_0 + \beta_1x_k + u_{0j} + u_{1j}x_k + e_{ijk}, $$ where the random intercepts have variance $\sigma^2_{u_0}$, the random slopes have variance $\sigma^2_{u_1}$, the random intercepts and slopes have covariance $\sigma_{u_{01}}$, and the residuals $e_{ij}$ have variance $\sigma^2_e$.

So what is the "repeatability of an effect" under this model? I think a good candidate definition is that it is the expected correlation between two pairs of difference scores computed within the same $j$ cluster, but across different pairs of observations $i$.

So the pair of difference scores in question would be (remember that we assumed $x$ is contrast coded so that $|x_1|=|x_2|=x$): $$ y_{i_1jk_2}-y_{i_1jk_1}=(\beta_0-\beta_0)+\beta_1(x_{k_2}-x_{k_1})+(u_{0j}-u_{0j})+u_{1j}(x_{k_2}-x_{k_1})+(e_{i_1jk_2}-e_{i_1jk_1}) \\=2x\beta_1+2xu_{1j}+e_{i_1jk_2}-e_{i_1jk_1} $$ and $$ y_{i_2jk_2}-y_{i_2jk_1}=2x\beta_1+2xu_{1j}+e_{i_2jk_2}-e_{i_2jk_1}. $$

Plugging these into the correlation formula gives us $$ ICC = \frac{cov(2x\beta_1+2xu_{1j}+e_{i_1jk_2}-e_{i_1jk_1}, 2x\beta_1+2xu_{1j}+e_{i_2jk_2}-e_{i_2jk_1})}{\sqrt{var(2x\beta_1+2xu_{1j}+e_{i_1jk_2}-e_{i_1jk_1})var(2x\beta_1+2xu_{1j}+e_{i_2jk_2}-e_{i_2jk_1})}}, $$ which simplifies down to $$ ICC = \frac{2x^2\sigma^2_{u_1}}{2x^2\sigma^2_{u_1} + \sigma^2_e}. $$ Notice that the ICC is technically a function of $x$! However, in this case $x$ can only take 2 possible values, and the ICC is identical at both of these values.

As you can see, this is very similar to the repeatability coefficient that you proposed in your question, the only difference is that the random slope variance must be appropriately scaled if the expression is to be interpreted as an ICC or "unadjusted repeatability coefficient." The expression that you wrote works in the special case where the $x$ predictor is coded $\pm\frac{1}{\sqrt{2}}$, but not in general.

(2.) When I have multiple variables whose repeatability I want to estimate, adding them all to the same fit (e.g. lmer(dv~(iv1+iv2|unit)+iv1+iv2) seems to yield higher repeatability estimates than creating a separate model for each effect. This makes sense computationally to me, as inclusion of multiple effects will tend to decrease the residual variance, but I'm not positive that the resulting repeatability estimates are valid. Are they?

I believe that working through a similar derivation as presented above for a model with multiple predictors with their own random slopes would show that the repeatability coefficient above would still be valid, except for the added complication that the difference scores we are conceptually interested in would now have a slightly different definition: namely, we are interested in the expected correlation of the differences between adjusted means after controlling for the other predictors in the model.

If the other predictors are orthogonal to the predictor of interest (as in, e.g., a balanced experiment), I would think the ICC / repeatability coefficient elaborated above should work without any modification. If they are not orthogonal then you would need to modify the formula to take account of this, which could get complicated, but hopefully my answer has given some hints about what that might look like.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.