Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

It's usually straightforward to do a Power Analysis to compute the minimum sample size, especially in R which is my preferred statistical computing environment.

However, I am being asked to conduct a Power Analysis that's a little different than anything I've done or which I can find reference to online. I'm wondering if what I'm being asked for is even possible/valid.

The project basically has two unequal groups of states and the hypothesis is that these two groups are significantly different in terms of an outcome variable (which is the duration of phone calls to customers). The "control" group consists of 40 states and produced about 2,500 observations. The "test" group has about 10 states and 500 observations.

Initially, I found group means + pooled standard deviation, which I used to calculate an Effect Size. Then I used a package called pwr in R and found that I needed a minimum sample size of about 135 observations per group, given .05 significance and .8 power.

However, they rejected my answer because they want one group to be much bigger than the other like it is now, and they are expecting either two different minimum numbers of observations per group or a minimum % of the population in terms of numbers of states or observations that have to go into their "test" group.

I see Power Analyses for two sample t-tests (i.e. the R function pwr.t2n.test), but I'd have to specify at least one of the sample sizes whereas they want me to tell them the minimal sample size for both groups (either as numbers or percentages) and this function doesn't reflect the differences in standard deviations for the two groups.

Is this possible or do I just tell them that's not how it works (i.e. the best I can do is tell them that given one of the sample sizes and a pooled standard deviation the second group has to be at least a certain size)?

share|improve this question

2 Answers 2

up vote 4 down vote accepted

You can do sample size calculations for unequal sample sizes.

For example, you can decide the n's are in some ratio (such as in proportion to the populations perhaps).

It's then possible to do power calculations (at the least you can simulate to obtain the power under any particular set of circumstances, whether or not you are able to do the algebra).

The problem is that it's relatively inefficient at finding differences compared to the same total number of observations at equal sample sizes.

Imagine you had a total sample of $n=n_1 + n_2$, with equal variance in the population and close to equal sample variance, and that your choice was between a 50-50 split and a 90-10 split ($n_1 = 0.5n$ vs $n_1=0.9n$).

The two-sample t-statistic is:

$t = \frac{\bar {X}_1 - \bar{X}_2}{s_{\text{pooled}} \cdot \sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$

The impact of the sample size is in the term $1/{\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$.

If you have the 50-50 split it's like having a 40% smaller standard deviation; you can pick up an effect that's 60% as large with the even split.

If the combined sample size is not an effective constraint, this calculation may pointless however. It matters in cases where every observation carries the same marginal cost, which is not always relevant.

share|improve this answer
    
Thanks! So, to actually carry this out is my best bet to simply use something like pwr.t2n.test() in R for basically every possible combination given a combined sample size, or is there a better way you'd recommend to compute this? If you use another language/package then I can figure out how to translate your answer into R. In either case, thanks for your solution. –  Hack-R Jul 15 at 20:29
    
I don't understand what you're getting at here. As far as I can tell you don't have the ability to split in any ratio you like; my example was to illustrate the effect of uneven splits. If you do have a fixed total sample size, then split it as near to 50-50 as would be accepted. If you don't have a fixed total sample size, then you optimize based on however your costs and constraints actually do go. –  Glen_b Jul 15 at 20:33
    
The extent of my information is that they currently have those two really unequal groups which total to 3,000 observations across 50 US states. They ask for the minimum number of observations per group and/or the % which should go into each group, based on the means and standard deviations I get from the data. I have no idea if they could split the data 50-50 or any way they like, but I was going to show them the minimum n in group 2 for every value of group 1's n and also show them the cost of splits other than 50-50, probably in terms of power. I thought that was roughly what you suggested? –  Hack-R Jul 15 at 20:43
    
I see. You could certainly calculate something like that, but you wouldn't necessarily need to do it at every $n$; these things progress quite smoothly. –  Glen_b Jul 15 at 20:46
1  
@Glen: That is why I tried to quickly remove my comment about the (un)usefulness of a t test in duration analysis. But you were even faster. Another concern is the grouped structure of the data. But your answer exactly fits to the question. –  Michael Mayer Jul 15 at 21:32

First off, why are you assuming equal variances in the two groups? Please don't say, "Because it's convenient." I seriously doubt that the group variances are equal, although in the case of equal sample sizes that isn't crucial. Your degrees of freedom will be off, but you know you have at least 130, so who cares? There are much bigger questions to address.

If you are going to permit (or require) unequal group sample sizes, the problem will not have a unique solution. There are two unknowns ($n_1$ and $n_2$ and only one constraint (the power must be at least $\phi$.) I don't think the problem can be solved without an additional constraint. There are two obvious possibilities. The first is to fix one of the sample sizes (e.g., the sponsors want at least 300 observations from Group I). The other is to fix the ratio (e.g., because Group I is ten times the count of Group II, we want $n_1 = 10\, n_2$). Now proceed with your power analysis.

share|improve this answer
    
How did I assume that there were equal variances? I know there's not... that's why I was asking about the possibility of using the different standard deviations for each group to do a calculation, though the normal way of doing a Power Analysis is to use pooled standard deviations when you have more than 1 group. –  Hack-R Jul 16 at 13:08
    
@NerdLife: Pooled variance, which you said you used, amounts to assuming equal variances. The "normal" way of doing it is to assume that $n_1 = n_2$, and in that case pooling doesn't matter to the value of the test statistic. The OSL is affected because the degrees of freedom shift between$\n_1-1$ and $2\,(n_1 - 1)$. But those shifts are pretty minor above 30 or so df. –  Dennis Jul 16 at 21:19

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.