Combining Results of Simulation Replications (Random-Intercept Logit Models under Confounding)

Question

I've written some simulation code in R to learn about the behavior of a random-intercepts logit model under varying degrees of confounding. The simulated scenario is three points in time, two groups, no time-by-group interaction (not yet, anyway) and a single continuous variable that may act as a confounder of the association between the group variable and the dichotomous outcome.

You might think of this in terms of rural vs. urban (groups) with the outcome being owning a pickup truck (outcome) and the confounder is age. Or something along those lines. Both age and rural/urban would be associated with pickup truck ownership in my simulation and I can vary the differences in age between urban and rural.

Here's my question. I'm computing some measures of association, namely the risk ratio for the outcome at each of the three points in time. The model adjusts for age and fits a linear time trend in the logit of the outcome.

I have two options:

1) I can compute the probability of the outcome at each point in time within each simulation replication, take the mean across replications, then compute risk ratio (and certain other quantities) using those overall mean proportions.

2) I can compute the relative risk (as well as its components "true" and "bias" in presence of confounding) within each simulation replication and then average the measures (risk ratios and so forth) across the replicates.

I initially coded it up as Option 1 and the measures look fine. But now I'm thinking to move on to perhaps computing percentiles of the measures across the replications and that directs me toward Option 2.

Is the a standard/usual way to aggregate these things in simulation studies?

Answer 1

Bias is defined as the difference between the expected value of an estimator and its true value. In simulation studies, we estimate this expected value by taking the average of the parameter of interest across simulation replications, which is what you have described with option 2, and which is the standard practice in simulation studies.

The result of option 1 estimates the ratio of the multiplicative biases of the numerator and denominator of the risk ratio, not the bias of the risk ratio. A problem with this approach is that if the numerator and denominator are both biased by the same factor, that will not show up in your estimate of the bias. I have never seen anyone use this approach, and I do not recommend it.