How can I estimate the power of a MaxDiff experiment with covariates?

Question

I am running a MaxDiff experiment where there will be four different "arms". Essentially, a respondent will be assigned to one of four conditions, where they will be shown one of three messages (plus a control), and then they will take the MaxDiff battery (which will include 20 items).

My question has multiple components. I know how to create a MaxDiff design (without covariates) in R using choiceDes or AlgDesign. But I do not know how to include the arm of the experiment (which condition the respondent is assigned to) as a covariate in generating the design.

I could, however, assume that I replicated the design across all four arms. But I would still be stuck here because I don't know how to evaluate the power of the design (whether or not covariates are included)

So I don't know how to:

1. Generate a MaxDiff design with covariates (in this case, the arm of the experiment)
2. Evaluate the power of the design (conditional on sample size) of a MaxDiff experiment with covariates
3. I also don't even know how to evaluate the power of the design without covariates

The goal here is to know what the sample size must be to in order to detect changes in utilities for the attributes across the different arms. We can of course make assumptions as needed about the effect size, etc.

The package skpr seemed promising. I had thought that I could generate a MaxDiff design (e.g. using choiceDes), assume that I replicated the design across all four arms, and use eval_design_mc to evaluate the power of the design. However, this only takes gaussian, binomial, poisson, or exponential families as distributions, and the actual model here is multinomial, not binomial.

So... I'm a bit stuck.

If necessary, I could simulate a dataset (making assumptions about effect sizes and sample size) and estimate the model using simulated data using mlogit, and use the resulting standard errors as a guide, but I'm hoping that there is a simple rule of thumb.

I have been through all of the documentation and rules of thumb that I can find online, e.g. this comment which talks about rules of thumb for subgroup analysis (but does not give any indication why or how to assess the quality of the experiment) and the linked paper ("What is the right sample size for my MaxDiff study?" in this pdf, which only covers the case without subgroups/covariates).

I am hoping there is a (relatively) easy way of thinking about this that does not require simulating a dataset. Any help would be greatly appreciated!

Answer 1

You could eventually use a formulae for choice proportions as explained in Louviere et al's book ("Stated Choice Methods: Analysis and Applications") - It won't do a formal power computation. This formulae is for the estimation of choice proportions. Your case is slightly different because you only care about the determinants of the choices (i.e., weights of the different attributes) but it remains a reasonably good approximation. The Louviere et al's formulae has been used quite a lot in the "discrete choice experiment" literature (e.g., ask people to choose the best option among 2+). Depending on how you intend to analyse your choice data, this formulae should also be relevant for MaxDiff. Key here would be to assume that best and worst choices are independent and provide same info about the determinants of the choices - Under this critical assumption, the worst choices could be seen as data augmentation method (i.e. asking people to answer 2 questions per task = asking 1 question per task but multiplying number of tasks per 2). Following this idea, if you want to perform more formal sample size computation, you will need to look at NGENE software which allows computing different statistical efficiency measures (or at least take a look at the NGENE manual, it will explain how to compute the Sp-Efficiency measure => minimum sample size needed for each effect of interest).