Is response variable/dependent variable data required for simr simulation?

Question

I'm using simr for power analysis of a linear mixed effects model. Most examples I've seen use data from pilot studies for simulation, but I don't have any data yet. My questions are:

1. Is data for the dependent/response variable necessary when using simr for power analysis?
2. I've tried running simulations using a dataset without a dependent variable, and it seems to work. Is this approach valid?
3. If data for the dependent/response variable is indeed necessary, should I generate random values? If so, how should I approach this?

Below is my current code in case that helps.

library(dplyr)
library(tidyr)
library(simr)

# Set parameters
n_subjects <- 100
n_events <- 24

# retrieve counterbalance of schemas & condition to balance frequency
counterbal <- read.csv("counterbalance.csv")
conditions <- expand.grid(
  distance = c("close", "far"),
  direction = c("antecedent", "consequence")
)

# randomly assign counterbalance group to participants
get_subj_data <- function(subj_id) {
  temp <- data.frame(subject = rep(subj_id, n_events))
  temp$event <- 1:n_events
  counter <- sample(1:4, 1)
  temp$conditions <- counterbal[[paste0("Counter", counter)]]
  temp$distance <- conditions$distance[temp$conditions]
  temp$direction <- conditions$direction[temp$conditions]
  return(temp)
}

df <- data.frame()

for (i in 1:n_subjects) {
  subj_data <- get_subj_data(i)
  df <- rbind(df, subj_data)
}

# H1 - main effect T(consq) > T(ante)
# here specify effect size! Not coefficients!
fixed <- c(0.5, 0.5, 0.5, 0.5)
rand <- list(0.2, 0.1) # not sure what is this
res <- 2

model <- makeLmer(latency ~ direction * distance + (1|subject) + (1|event),
                  fixef=fixed, VarCorr=rand, sigma=res, data=df)
model

# simulation on direction?
sim_direction <- powerSim(model, nsim=100, test = fcompare(~ distance))
sim_direction

Which gives me the following output:

> sim_direction
Power for model comparison, (95% confidence interval):
      100.0% (96.38, 100.0)

Test: Likelihood ratio
      Comparison to ~distance + [re]

Based on 100 simulations, (0 warnings, 0 errors)
alpha = 0.05, nrow = 2400

Time elapsed: 0 h 0 m 12 s
```

Answer 1

Welcome to CV :)

Is data for the dependent/response variable necessary when using simr for power analysis?

No, but in order to run a power analysis, the software requires effect sizes, for the fixed effects, random effects and residual variance. If the response variable is present it will estimate these from the data. If not, then these values should be provided - often obtained from previous studies, or from a pilot study.

I've tried running simulations using a dataset without a dependent variable, and it seems to work. Is this approach valid?

Yes, I think so; according to the comments to the question, you have provided the effect sizes.

If data for the dependent/response variable is indeed necessary, should I generate random values? If so, how should I approach this?

I think we have covered this above. Providing random responses would not be a good idea.

Lastly, please be aware that studies where repeated measures are made on subjects often require careful consideration of serial dependence of observations, which is typically not handled well by the default methods for fitting mixed effects models in most software. Many packages for fitting mixed effects models provide ways to handle this, but I doubt they are available in simr (I would be happy to be corrected on this).

Answer 2

A simr vignette says explicitly:

If pilot data is not available, simr can be used to create lme4 objects from scratch as a starting point. This requires more param[e]ters to be specified by the user. Values for these parameters might come from the literature or the user’s own knowledge and experience.

After you specify the data frame of predictor values, the fixed effects, the random effects and the residual variance, makeLmer() "generate[s] random values" for you based on those assumptions. You don't need to generate more on your own. Insofar as those specifications are close to reality, then the power estimates will be valid. If you just pulled those specifications out of the air, however, there's no assurance that the power estimates will work in practice.

I'm not sure that you specified the fixed effects correctly, however. As I understand the makeLmer() function, it uses the value of fixef to be the intercept and coefficients for the fixed effects. If those are what you intended, to correspond to the sigma=res value of 2 for the residual standard deviation in the outcome scale, that's fine. If not, reconsider how you specified those. For example, using sigma of 1 would put everything in a scale of residual standard deviation units. (I'm not sure which definition of "effect size" you are using.)

Your code suggests that you don't know what the rand value passed to the VarCorr argument represents. In general, that's a list of variance-covariance matrices for the random effects. As you only are using random intercepts, each of those matrices reduces to a single Gaussian variance. You specified 0.2 for the variance of subject and 0.1 for the variance of event. Were those what you intended?

Finally, be careful in how you set up the data data frame for the predictor-variable values. In practice, in something other than a completely designed experiment, those predictor values are typically correlated with each other. If you ignore that in setting up that data frame then your power estimates also might not be valid, as those correlations might increase the variances of the fixed-effect coefficients.