correct sample size using power analysis

Question

I have sample data below for 10 groups and I report the number of observations (count), mean and standard deviation below.

  group= 1:10
    count = c(41,640,1000,65,30,4010,222,277,1853,800 )
    mu = c(.7143,.66,.6441,.58,.7488,.5616,.5507,.5337,.5513,.5118)
    sd = c(.2443,.20,.2843,.2285,.2616,.2365,.2408,.2101,.2295,.1966)
    dat = data.frame(group= group, count = count, mu = mu, sd = sd)
    dat[order(dat$count),]

  group count     mu     sd
5      5    30 0.7488 0.2616
1      1    41 0.7143 0.2443
4      4    65 0.5800 0.2285
7      7   222 0.5507 0.2408
8      8   277 0.5337 0.2101
2      2   640 0.6600 0.2000
10    10   800 0.5118 0.1966
3      3  1000 0.6441 0.2843
9      9  1853 0.5513 0.2295
6      6  4010 0.5616 0.2365

Right now I am plotting the means with 95% confidence intervals and I don't think all the means are comparable due to sample size differences so I was thinking was to do power analysis on the 2 groups with the MOST and LEAST number of observations (GROUPS 5 & 6) to a number of observation threshold.

So I do power analysis using Groups 5 & 6 means and sds

library(pwr)
pwr.t.test(d=(.5616-.7488)/ sqrt((.2365^2+.2616^2))/2,power=.8,sig.level=.1,type="two.sample",alternative="two.sided")


  Two-sample t test power calculation 
              n = 176.201
              d = 0.2654139
      sig.level = 0.1
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

The result says I need 176 observations in each group.

From here I would state: GROUPS 5,1, and 4 need more data before they can considered in any comparison because they have low sample size.

Questions:

(1) Is that the correct interpretation

(2) Is this the correct methodology to follow. I.e. I Chose groups 5 & 6 because of the differences in sample size BUT the "n" returned from the power analysis does not consider the sample size. SO -- if the effect size was very small even though the samples sizes differed a lot I would have gotten a very large "n" as my threshold.

as an example if group 5 had a mean of .64 insted of .74

library(pwr)
pwr.t.test(d=(.5616-.64)/ sqrt((.2365^2+.2616^2))/2,power=.8,sig.level=.1,type="two.sample",alternative="two.sided")


Two-sample t test power calculation 

          n = 1001.387
          d = 0.1111562
  sig.level = 0.1
      power = 0.8
alternative = two.sided

"n" is now 1000 instead of 176.... so when comparing multiple groups what is the correct way to determine which groups need more observations?

I can arbitrarily decide and say I need 100 observations but I was thinking power analysis would help.

NOTE:

The only data I am given is what is in the table. I do not have the underlying data to run are regression or do pair wise tests. I am comparing the means of groups in the table but I think the comparison is inaccurate as is because of major sample size differences.

ALSO: in response to " what you wish to accomplish with your data and why you are trying to define "large sample."

When I am comparing means in the table above some groups will have larger sample sizes leading to narrower confidence intervals while other groups with very low sample size will have large error bars. I am hoping to identify the optimal sample size. I.e. Given there are multiple groups with different sample sizes from 30 to 4000 should I set a threshold that says "I need more data from group X, Y,... because the average is so uncertain" and if so how to determine that threshold.

Answer 1

Power analysis is something you do when designing a study, not after you have collected the data.

Your data seem amenable to a simple linear model (ANOVA) comparing the groups, as the standard deviations seem very similar among the groups. With large differences in numbers of cases the standard errors and confidence intervals will differ dramatically among groups, but the mean values are nevertheless comparable among groups. You just need to take the number of cases into account when you compare among groups.

A proper linear model will share information about the residual errors among the groups more usefully than depending on SDs of individual groups, as you seem to be doing. Testing any pre-defined hypotheses about inter-group differences, or against a pre-defined threshold, would then be straightforward, and you can do general post-hoc tests of differences among groups, although the large differences in cases among groups might make some approaches to post-hoc analyses challenging.

Even though you do not have the underlying raw data, the data in your table contain all that you need to do these analyses. The squared standard deviations about each of the mean values provide estimates of the residual sums of squares that can be pooled to provide an overall mean-square residual estimate. Overall ANOVA compares the variance among group means against the residual error. Comparisons among particular means then use that mean-square residual and the number of cases in each group in the equivalent of t-tests (with appropriate correction for multiple comparisons if you are not examining pre-defined hypotheses). This is pretty much how ANOVA was done with hand calculators in the "good old days" before computers and canned programs. You will, however, be limited in your ability to perform tests for outliers and leverage without the individual data points.

You also might consider the way that you defined the 10 groups. You might consider re-defining the groups in a way that combines some of the lower-count groups, if that makes sense based on your knowledge of the subject matter.

If you are using these data as a pilot guide to further data collection for studies of planned comparisons, then all you need is the pooled mean-square error estimate, and the outcome difference that you wish to be able to detect with a particular power at a particular significance level, as in your use of the pwr package. That should, however, best be a pre-determined difference, not the observed difference in this set of data. It can be risky to go back and just sample particular groups that seem to be too small in your data set to find some hoped-for "significant" difference between 2 groups. The power calculations are not designed to take such result-chasing into account. Also, you need to correct for multiple comparisons if you are not testing a small number of pre-planned (i.e., before you saw these data) hypotheses.