Let's say I run an experiment where participants play a behavioral game with someone that is either their same gender, or not their own gender. The primary criteria of interest is a pre and post measure of some outcome (taken before and after the game), captured as a change-score.

My approach for modeling this looks like the following:

 felm(change_score ~ my_gender + other_gender_pairing + my_gender:other_gender_pairing | team_id | 0 | 0) 

Where other_gender_pairing is a dummy variable set to 1 if you're paired with someone of the opposite gender. Standard errors are clustered a the team_id level, since all dyads will have correlated error terms.

I have already conducted a pilot of this experiment involving 80 participants in each "cluster." Where a 0 or 1 indicates whether somebody was paired with their same gender (M-M or F-F = 0) or opposite gender (M-F or F-M = 1)

              0  1
  Male       80 80
  Female     80 80

I would like to conduct power analysis to figure out the size of the effects I can detect given some fixed sample size per cell, informed by the coefficients estimated in the pilot. Unfortunately, because this design is somewhat complicated by the interaction effect and clustered standard errors, I cannot find any useful guides.

Can somebody point me in the right direction?

For reference, here are the coefficients identified in the regression run on the pilot. I'd like to be able to incorporate these estimates for more accurate power analyses.

                                                Estimate Std. Error t value  Pr(>|t|) CI Lower CI Upper  DF
(Intercept)                                      -0.8254      1.169 -0.7061 0.4817274   -3.144    1.493 101
my_gender                                         3.1266      2.092  1.4947 0.1381022   -1.023    7.276 101
other_gender_pairing                             11.6639      3.330  3.5023 0.0006886    5.057   18.270 101
my_gender:other_gender_pairing                   -5.0025      4.645 -1.0769 0.2840919  -14.218    4.213 101

I use post-estimation tools to estimate contrasts where I compare means of M:M to M:F and F:F to F:M. This results in the following:

 contrast estimate   SE  df t.ratio p.value
 0 - 1      -11.66 3.33 199 -3.502  0.0006 

gender = Female:
 contrast estimate   SE  df t.ratio p.value
 0 - 1       -6.66 3.27 199 -2.040  0.0427 

I'd like to know how big of a sample size to have in each cell in order to detect effects for the interaction effect.

Edit: I should note that my preferred language is R. It would be very helpful to see some template code for how to deal with experimental designs like this.

  • $\begingroup$ Is the game symmetric? Do players ever play again with a different partner? $\endgroup$ – eric_kernfeld Dec 5 '19 at 23:32
  • $\begingroup$ Is there one change-score per person per game, or one per dyad per game? $\endgroup$ – eric_kernfeld Dec 5 '19 at 23:33
  • $\begingroup$ Players only play the game once. Think of it like a collaborative jigsaw puzzle. The change score is one person per game. All measures are at the individual level $\endgroup$ – Parseltongue Dec 7 '19 at 0:08
  • $\begingroup$ Are you looking for an analytical solution, or could you just run simulations? $\endgroup$ – eric_kernfeld Dec 8 '19 at 3:22
  • $\begingroup$ What is the difference between "MF" and "FM"? $\endgroup$ – eric_kernfeld Dec 8 '19 at 3:24

Since players only play the game once, dyads can reasonably be assumed to be independent of one another. It could potentially simplify your analysis, while maintaining your goals, if you worked with "one observation = one dyad" rather than "one observation = one person." You could dispense with the clustered errors and just estimate your interaction directly as

$$(\mu_{MM} - \mu_{MF}) - (\mu_{FF} - \mu_{FM})$$

where $\mu_{MF}$ is the mean over all change-scores of males paired with females. The possible issue here is that, depending on how your notation is understood, $\mu_{MF}$ and $\mu_{FM}$ are not statistically independent: they are the same set of games, just with the opposite person measured. That would sink this solution and then you'd be back to the drawing board.

  • $\begingroup$ You are correct with your last sentence, Eric. The mean of MF and FM are the same set of games, just with the opposite person measured. And therein lies the trouble with power analysis simulations. Any ideas? $\endgroup$ – Parseltongue Dec 9 '19 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.