0
$\begingroup$

I using simulated data sets to compare the ability to two different experimental designs (A and B) to detect a interaction between two variables (x and y) in determining the observed output (o). The results appear contradictory to me, and I think there must be an aspect of how lmer works that I'm not understanding that can explain this.

I am using lmer from the lmerTest package in r to fit the stimulated data to a model, and calculate the coefficient, SE of the coefficient, and p-value:

fit<-lmer(O~x+y+x*y+(1|subjectID),data)
c<-summary(fit)$coefficients
p<-c['x:yTRUE','Pr(>|t|)']
beta<-c['x:yTRUE','Estimate']
betaSE<-c['x:yTRUE','Std. Error']

When I generate ~500 replicates of simulated data for each trial design (A and B), and run each replicate through this analysis, I find:

Design A has higher power (a larger fraction of the replicates find a statistically significant effect) and lower mean betaSE (as defined above), despite the fact that Design B has a similar magnitude of mean beta, a significantly smaller standard deviation of the beta (SD calculated across the replicates).

What should this be telling me about why Design B has lower power to detect a significant effect?

EDIT: Designs A and B are two different clinical trial designs where participants spend different amounts of time on open label drug and blinded drug vs placebo. The simulated data are created with a complex set of flexible assumptions about the relationships between 3 factors in a model of symptom burden over time as a function of time, expectancy and drug response. These assumptions are used to create a covariance matrix, which is then used by mvrnorm to generate simulated factor-level data as per the specified covariance matrix. The factors are summed to create simulated data that represents what would actually be produced by a clinical trial. This data is what is analyzed using the above methods.

$\endgroup$
2
  • $\begingroup$ Please provide some information about how trial designs A and B differ and how you generated the simulated data sets for those designs. $\endgroup$
    – EdM
    Feb 17, 2020 at 18:52
  • $\begingroup$ The designs/simulations are complex, but I have attempted to add what seems like the most relevant information above... thanks for considering the question! $\endgroup$
    – neuropsych
    Feb 17, 2020 at 20:15

1 Answer 1

0
$\begingroup$

Well, this should have been obvious: I have a whopping high type 1 error rate in design A. Lmertest is using the Satterthwaite method, but somehow in this context it is producing a high false positive rate for one design and not for the other.

If I figure out why, I'll add it here.

EDIT: It's because even when the model fit isn't actually throwing a singularity error, the correlation structure of the data from design A was close enough to singular to be inflating the type I error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.