# Calculating sample size with power analysis

I have to calculate the sample size for an experimental study and took some sample distributions data from a previous similar research and calculated the means and standard variations for before and after the treatment (see picture).

1. since the sample distribution is not normally distributed, does it still make sense to calculate the standard deviation?

2. is it common to use the effect size from previous studies or should one even lower the effect size for sample size calculation?

3. how do I process further in order to get a sample size, whit power of 80%, alpha .05?

I habe never done a power analysis before, so can anyone help?

1. The outcomes represent the payouts that come with the reports done by the participants. In many previous papers, the means were just compared.

2. Okay, good to know! But how does one come up with the answer? Do I have to look for papers, where no significant difference was found?

3. I was actually going try to compare the means, but since the outcome distributions are not normally distributed I guess one has to use non parametric test. It would be the wilcoxon rank sum test then?

1. Standard deviation applies to non-normally distributed data. Since the outcomes are categorical, their numerical labelling are more or less arbitrary. I could rename them to inflate/deflate the variance at will. Hence, it might be easier to look at the standard deviation of a single outcome, or to look at the multinomial covariance matrix.

2. Depends on the quality of the study. My own advice is to select a smallest meaningful effect size and base your study on that. Ask yourself "what effect size is small enough but is still interesting to you?". Another way to ask that would be "At what point would the effect be too small to matter to you?".

3. Power analysis depends on the analysis you intend to perform. From the pictures you include, it looks like you're doing a chisquare test. Is that still what you intend to do?

Understanding current data. It looks as if the counts from your previous experiment may have been something like those in the table below, where the first row is for the control group and the second for the treatment group, and columns are for 0 through 5.

TAB
[,1] [,2] [,3] [,4] [,5] [,6]
n.c   10   11    9   10    8    8
n.t   16    4    5    9   21   30


A chi-squared test would have rejected the null hypothesis that scores 0-5 are independent of group, with P-value 0.002.

chisq.test(TAB)

Pearson's Chi-squared test

data:  TAB
X-squared = 19.261, df = 5, p-value = 0.001718


But this does not say that scores are higher in the treatment group, only that the distributions differ--as one can readily see from the histograms your posted.

chisq.test(TAB)

Pearson's Chi-squared test

data:  TAB
X-squared = 19.261, df = 5, p-value = 0.001718


The lists of individual scores for treatment and control groups can be reconstructed as follows:

x.t = rep(0:5, times=n.t)
table(x.t)
x.t
0  1  2  3  4  5
16  4  5  9 21 30
summary(x.t)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.000   2.000   4.000   3.235   5.000   5.000

x.c = rep(0:5, times=n.c)
table(x.c)
x.c
0  1  2  3  4  5
10 11  9 10  8  8
summary(x.c)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.000   1.000   2.000   2.339   4.000   5.000


The treatment mean and median are larger than the control mean and median. A Wilcoxon rank sum test confirms that the treatment scores are centered above the control scores:

wilcox.test(x.t, x.c, alt="greater")

Wilcoxon rank sum test
with continuity correction

data:  x.t and x.c
W = 3068.5, p-value = 0.001552
alternative hypothesis:
true location shift is greater than 0


One can quibble about what a 2-sample Wilcoxon test means when the two samples have different shapes. However, empirical CDF plots show that the treatment scores (blue) 'stochastically dominate' the control scores; that is, the treatment ECDF is (mainly) to the right of the control ECDF (thus below it on the cumulative probability graph).

plot(ecdf(x.c), col="brown",
main="ECDF Plots of Data (Control in Brown)")
lines(ecdf(x.t), col="blue")


Power and sample size. From the one prior experiment of this kind we can guess that having about 84 subjects in the treatment group and about 56 in the control group, gives enough power to detect a difference between the two groups.

If the control group for a subsequent experiment will give nearly a uniform distribution of the six scores and you have some idea what distribution among the treatment group you want to detect, then we can do a simulation to approximate the power. It is a little more efficient to have equal sample sizes in the treatment and control groups, so I will start with $$n_t = n_c = 75.$$ Also, I will suppose we want to detect a treatment distribution with probabilities pr $$= (1,1,2,2,3,3)/12.$$ respectively, for the six values $$0$$ through $$5.$$ For this particular scenario, you will have power about 78%.

set.seed(112)
pr = c(1,1,2,2,3,3);  n = 75
pv = replicate(10^5, wilcox.test(sample(0:5,n,rep=T),
sample(0:5,n,rep=T,p=pr),alt="less")$p.val) mean(pv <=.05) [1] 0.77812  If you need to detect a less extreme pattern or probabilities, then the power would be smaller; you would reject against this less extreme alternative more than half the time with 75 subjects in each group. set.seed(112) pr = c(1,1,1,2,2,2); n = 75 pv = replicate(10^5, wilcox.test(sample(0:5,n,rep=T), sample(0:5,n,rep=T,p=pr),alt="less")$p.val)
mean(pv <=.05)
[1] 0.55869


This smaller effect is more often detected when you have 135 subjects in each group; power about 78%.

  set.seed(112)
pr = c(1,1,1,2,2,2);  n = 135
pv = replicate(10^5,wilcox.test(sample(0:5,n,rep=T),
sample(0:5,n,rep=T,p=pr),alt="less")\$p.val)
mean(pv <=.05)
[1] 0.779


I don't know any closed-form power formulas for such Wilcoxon tests, but you can get a reasonably good idea what sample sizes to use if you can guess the kind of difference you want to find.

• Thank you for your detailed calculations and explanations! Wow! But you used different frequencies at the beginning, right? Jan 18, 2021 at 7:17
• Could you explain a bit more about the intuition behind the probabilities for the treatment distribution outcomes? Jan 18, 2021 at 7:22
• For our designed experiment we would take the effect size of the previous experiment as the estimated effect size (or even higher), and would use the common 80% power and 0.05 alpha. Though I am not really sure how to calculate an estimate for the standard deviation with the information given. I have one standard deviation for each group that differ. Jan 18, 2021 at 7:26
• The standard deviation is not of much use for the distinctly on0normal and discrete distributions you are using. Also I know of no way to define effect size for your distributions. I have given you several examples how to find power. // For normal distributions the SD is useful. Jan 18, 2021 at 9:45
• You did not explicitly show the frequencies you had in the current expt. I tried to deduce them from your histograms. // Also I used proportions (1,1,2,2,3,3) for first power simulation because it seemed you wanted higher counts for higher numbers 0 through 5. (You can choose any 6 numbers you want.) // In R, it is OK to use p =c(1,1,2,2,3,3) because R supplies denominator 12 to turn the proportions into probabilities. Jan 18, 2021 at 10:41