Power analysis for unknown experiment duration

Question

How would I do a power analysis if I do not know how long we want to run the study?

I want to use analysis to inform how long I want to run the experiment based on how long it will take to gather the samples needed. The part that I do not understand is - if I do not know how long we want to run the experiment, how do I come up with the mean and standard deviation needed to plug into power calculation since mean and sd will change over time? For example, if I want to measure sales per user, multiple sales can happen over time.

Answer 1

I would recommend that you determine your sample size by simulation, similar to this earlier thread.

Simulate "reasonable" data. Since you are looking at sales, take a look at historical sales time series to get an idea of possible distributions or influencing factors, like seasonality or promotions. Perhaps resample existing sales time series, with or without modification.

Then include an assumption about the size of the effect you want to detect with your experiment. My recommendation about what size to assume is always to use an effect you would be sorry to miss. If you are looking at the sales uplift from a changed store layout, maybe an uplift of 2% would be too small to roll out the changed layout to all stores, but an uplift of 5% would be worth it. If so, modify your simulated sales in the experimental group by 5%, e.g., by simulating negative binomial sales with a mean that is 5% higher.

Run your "experiment" for a given time period, say a month. "Analyze" your data the way you plan, and see whether you find the simulated effect. Do this multiple (many) times and see, in fact, whether you find the effect as often as you want to. A power of 80% (i.e., finding the effect in 80% of the simulations) is often targeted, but you may aim for a higher or a lower power.

If your power is too low, increase the sample size, by increasing the length of the "experiment" (or including more stores if you have a setup as I allude to above, or use more data in some other way appropriate to your experiment). If your power is too high, you can likely run your experiment with less resources, by reducing the sample size.

A setup like this allows you to test various assumptions. Perhaps some of your experimental units may drop out - you can simulate this. Maybe you have predictors - include them (and assumptions about their distribution, and their influence on your target variable) in your data generating process and your analysis plan.

Yes, this is quite some work, but you can timebox it and only be as elaborate in your simulation as your time for sample size determination allows. It's still typically better to invest a few days in this planning step, rather than find out afterwards that your study was under- or overpowered.

Answer 2

This is an important issue for metrics that accumulate over time, like sessions or revenue per user.

You need to use your historical data to see how the number of unique users N, the mean m1 of the user-level outcome, and the standard deviation sd of the user-level outcome increase with the duration of the test. You update these by calculating the running version of these three by day. You will also need to adjust the data if there are strong trends or seasonality around the time of the test. You don't want to assume that you get your peak traffic in designing the test if conditions during the test will be worse.

Once you have your three columns, you do the usual minimum reliably-detectable effect (MRDE) calculation where N, m1, and sd move in tandem rather than sending N to infinity while keeping m1 and sd fixed.

This returns something like this with a simulated dataset but real MRDE calculations for a two-sided t-test:

Here you get ~5K users on day 1, with a mean of 17.27 and a standard deviation of 19.26. Splitting them into equal experiment groups and assuming sd1=sd2=sd, you can detect a delta of 1.51 (in levels) or 9%.

On day 2, an additional 15K users show up for a total of 20K. The updated mean is now higher, and so is the sd. The delta is now 1.73, but in percent, that is 6%, so that's an improvement.

On day 3, you only get 7.7K new users, so the MRDE does not drop that much.

You keep going until you find an MRDE that you can live with. That pins down the approximate test duration. To detect a 4% lift, you need at least a 17-day test with my simulated data. I would pad that by a few to be a bit conservative.