1
$\begingroup$

I am wondering if it should be possible to check if random effects should be included in a mixed model based on soundness (i.e. using ecological reasoning). The general idea is, that it should hopefully be possible to look at between subject variances and compare them to known values to see if a model without random effects would even make sense.

I don't have a dataset as an example, but I still would like to know if my hypothetical reasoning is correct. Let's just assume I have a bunch of reaction times (in reality not normal distributed, but we will just assume normality here) from various participants, each sampled 100 times. Now also assume, that I run a mixed model with a random intercept and I get a standard deviation of $\sigma_{between}=1000ms$ for the Intercept. As far as I understand, this means, that the intercept varies 1000ms between participants.

Lets call the $i$-th measurement from the $j$-th participant $X_{i,j}$ and the mean for participant $j$ be $\bar{X_j}$.

Now the idea I had was to check if an alternative H0 could be tested, that all participants have the same mean and the variation between participants I am observing is purely by chance. However, depending on the variance assumed under H0, I could observe any between subject variance, so there is nothing to test statistically here. What I could calculate, however, is the sd given H0. If all subsets are purely random around the same mean with an sd of $\sigma_0$ under H0, that would imply that the mean of random disjoint subsets of 100 Measurements would be distributed with the same mean and a variance of $\sigma_0^2/100$. Since we know the means of the sets to be distributed with $\sigma_{between}=1000ms$ we can solve $\sigma_0^2/100=(1000ms)^2$ and therefore $\sigma_0=1000ms*10=10s$

Now based on that, I may just conclude that a standard deviation of 10s between measurements is simply much too high to be true and therefore a random intercept must be included.

However, I am not sure that this reasoning is sound at all. Also, I can easily do the calculation of the variance for an intercept, because it relates to the mean. If this method makes sense at all, it would be interesting if a similar method could also be applied for random slopes.

NOTE:

If this reasoning makes sense, it would not be enough to simply estimate the sd or variance across the sample, because if H0 does not hold, the measurements are correlated, giving reduced estimates of the real (within subject) variance.

Code to test this idea

library(dplyr)

# Reproducibility
set.seed(327401)

N <- 100 # number of measurements
M <- 100 # number of participants / subsets

mu <- 0
sigma <- 10

# Generate some random data

df <- expand.grid(id=1:M, measurement=1:N)
df$value <- rnorm(nrow(df), mu, sigma)

df %>% group_by(id) %>% summarize(MeanOfSet=mean(value)) %>% 
   summarize(MeanOfMeans=mean(MeanOfSet), 
             SdOfMeans=sd(MeanOfSet), 
             VarOfMeans=var(MeanOfSet))

# Gives: 
# MeanOfMeans SdOfMeans VarOfMeans
# -0.05118621 0.9692551 0.9394555
# i.e. the Sd and Var is reduced as expected

## Same test, but now with between subject variances around the
## same mean

sigma.between <- 5

df <- data.frame(id=1:M, intercept=rnorm(M,mu,sigma.between))
df <- merge(df, expand.grid(id=1:M, measurement=1:N), by="id")
df$value <- rnorm(nrow(df), 0, sigma) + df$intercept

df %>% group_by(id) %>% summarize(MeanOfSet=mean(value)) %>% 
       summarize(MeanOfMeans=mean(MeanOfSet), 
                 SdOfMeans=sd(MeanOfSet), 
                 VarOfMeans=var(MeanOfSet))
# Gives:
#   MeanOfMeans SdOfMeans VarOfMeans
#   -0.1991563  5.317212   28.27275
# i.e. the expected between subject variances

df %>% summarize(Sd=sd(value), Var=var(value))
# Gives:
# Sd       Var
# 11.29574 127.5936
# However, to see the given between subject variances by
# chance, sd should rather be 50
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.