1
$\begingroup$

I am designing an experiment that tests a difference in means between two groups in which data points take very long to generate (1 month each). Accordingly, I'd like to run several tests of the same hypothesis in parallel and combine the results. How can I determine the sample size needed for each group?

For example, if I ran one experiment I would need n=13 samples in each of the two groups based on the true difference in means I'm hoping to detect, the sd, power, and sig. level.

If I can instead run 2 or 3 experiments in parallel testing the same hypothesis on distinct datasets (where the datasets have similar sds but different baseline means), what is the sample size needed in each group to achieve the same power and sig. level as the n=13 in one experiment testing the same hypothesis?

Experiment A: compare means between group A_1 and group A_2

Experiment B: compare means between group B_1 and group B_2

Experiment C: compare means between group C_2 and group C_2

Then combine results.

I realize if I had p-values from multiple tests it would be a meta-analysis problem (e.g., Stouffer's Method), but I'm not sure how to determine the sample size needed in the first place for a set power and significance level. Thanks!

$\endgroup$
  • $\begingroup$ Why run the tests in parallel? If the data are from the same phenomenon and the same groups, why not aggregate the data? $\endgroup$ – Demetri Pananos Feb 1 '19 at 6:45
  • $\begingroup$ Edited - the baseline mean in each experiment may be different. Imagine I'm trying to test an improved engine in a group of busses. In one experiment the busses go from A to B in a repeated way, while in the second experiment another group of busses go from C to D in a repeated way. So the baseline mean could be different between experiment 1 and 2 depending on the length of the roads traveled, but in both cases we're testing the efficacy of the new engine. $\endgroup$ – Cardinal2016 Feb 1 '19 at 6:54
  • $\begingroup$ Is it possible to frame your problem as a regression? Are you altering something in each experiment which could be considered a covariate? Can you provide some actual context so that we aren't left guessing? $\endgroup$ – Demetri Pananos Feb 1 '19 at 6:57
  • $\begingroup$ I'm trying to test if a new system of dynamic routing for long-haul trucks is more fuel-efficient. So for an origin-destination pair, one set of trucks will take the typical route as before while the second set will follow a dynamic route. Each datapoint is then total fuel consumed on one journey. We then compare the means of fuel consumption of the two groups, but it requires too many journeys, so I'm trying to instead test across multiple origin-destination pairs. Thanks for your help. $\endgroup$ – Cardinal2016 Feb 1 '19 at 7:07
  • $\begingroup$ OK, thank you that is very helpful. Is it at all possible to have the same number of trucks in test and control, or is it prohibitive to have trucks in the test (dynamic routing) group? $\endgroup$ – Demetri Pananos Feb 1 '19 at 7:09
1
$\begingroup$

I am going to suggest something a little different.

The fuel saved will depend on the route length. You should control for that somehow, else the effect estimates will be different in the three groups (larger in groups where the route is longer, and smaller in groups where the rout is shorter).

I'm going to suggest you do a linear regression, something like

$$\text{Fuel Consumed} = \beta_1\text{Route Length}\cdot(1+\beta_2\text{Group}) $$

Here, the group covariate is a binary variable (0 for control, 1 for test). The intercept here is zero because I assume if the route length is 0 miles, then the fuel consumed is 0. But you didn't ask for methods of analysis, you asked for sample size calculations.

For a linear regression, the sample size required for a minimial detectable effect of $\beta_j$ at power $\gamma$ is

$$ n = \dfrac{(z_{1-\alpha/2}+z_{\gamma})^2 \sigma^2_{y\vert x}}{(\beta_j\sigma_{x})^2(1-\rho_j^2)} $$

Here, $\sigma^2_{y \vert x}$ is the residual error from the linear regression, $\beta_j$ is the minimal detectable effect you want to find, $\sigma_x$ is the standard deviation of the predictor (which can be rescaled, so it doesn't really matter), and $1/(1-\rho^2)$ is the variance inflation factor. If you randomize trucks to test and control, then in principle $\rho_j^2 = 0$. The $z$ are standard normal quantiles at the indicated levels. You also indicate you may have access to past data, which you could maybe use to estimate $\sigma^2_{y \vert x}$.

My point is that you can estimate how many trucks you need in order to detect your minimal effect, and then split them into the three groups. Under this design, $\beta_2$ should be around -0.05 (as per your comment about expected fuel savings). Since you are designing the experiment, you are also free to set $\sigma_x$ by (hopefully) deciding which trucks on which routes will be part of the experiment.

This approach has the added benefit that all the data is analyzed at once, and only one test is computed (namely the t test for $\beta_2$).

Perhaps a simpler, equivalent, option would be to pair trucks (test and control), divide the fuel consumed by the test truck by the fuel consumed for the control truck, and then just do a t-test on the entire data set. Now you can use the t-test sample size computations with under the hypothesis that the true effect is -0.05.

It may be a good idea to perhaps add a random effect if the routes in each group are the same.

I'll let someone who is a little smarter and a little less sleep deprived comment and tell you if what I have proposed is a good idea or not.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.