I have sample data below for 10 groups and I report the number of observations (count), mean and standard deviation below.
group= 1:10
count = c(41,640,1000,65,30,4010,222,277,1853,800 )
mu = c(.7143,.66,.6441,.58,.7488,.5616,.5507,.5337,.5513,.5118)
sd = c(.2443,.20,.2843,.2285,.2616,.2365,.2408,.2101,.2295,.1966)
dat = data.frame(group= group, count = count, mu = mu, sd = sd)
dat[order(dat$count),]
group count mu sd
5 5 30 0.7488 0.2616
1 1 41 0.7143 0.2443
4 4 65 0.5800 0.2285
7 7 222 0.5507 0.2408
8 8 277 0.5337 0.2101
2 2 640 0.6600 0.2000
10 10 800 0.5118 0.1966
3 3 1000 0.6441 0.2843
9 9 1853 0.5513 0.2295
6 6 4010 0.5616 0.2365
Right now I am plotting the means with 95% confidence intervals and I don't think all the means are comparable due to sample size differences so I was thinking was to do power analysis on the 2 groups with the MOST and LEAST number of observations (GROUPS 5 & 6) to a number of observation threshold.
So I do power analysis using Groups 5 & 6 means and sds
library(pwr)
pwr.t.test(d=(.5616-.7488)/ sqrt((.2365^2+.2616^2))/2,power=.8,sig.level=.1,type="two.sample",alternative="two.sided")
Two-sample t test power calculation
n = 176.201
d = 0.2654139
sig.level = 0.1
power = 0.8
alternative = two.sided
NOTE: n is number in *each* group
The result says I need 176 observations in each group.
From here I would state: GROUPS 5,1, and 4 need more data before they can considered in any comparison because they have low sample size.
Questions:
(1) Is that the correct interpretation
(2) Is this the correct methodology to follow. I.e. I Chose groups 5 & 6 because of the differences in sample size BUT the "n" returned from the power analysis does not consider the sample size. SO -- if the effect size was very small even though the samples sizes differed a lot I would have gotten a very large "n" as my threshold.
as an example if group 5 had a mean of .64 insted of .74
library(pwr)
pwr.t.test(d=(.5616-.64)/ sqrt((.2365^2+.2616^2))/2,power=.8,sig.level=.1,type="two.sample",alternative="two.sided")
Two-sample t test power calculation
n = 1001.387
d = 0.1111562
sig.level = 0.1
power = 0.8
alternative = two.sided
"n" is now 1000 instead of 176.... so when comparing multiple groups what is the correct way to determine which groups need more observations?
I can arbitrarily decide and say I need 100 observations but I was thinking power analysis would help.
NOTE:
The only data I am given is what is in the table. I do not have the underlying data to run are regression or do pair wise tests. I am comparing the means of groups in the table but I think the comparison is inaccurate as is because of major sample size differences.
ALSO: in response to " what you wish to accomplish with your data and why you are trying to define "large sample."
When I am comparing means in the table above some groups will have larger sample sizes leading to narrower confidence intervals while other groups with very low sample size will have large error bars. I am hoping to identify the optimal sample size. I.e. Given there are multiple groups with different sample sizes from 30 to 4000 should I set a threshold that says "I need more data from group X, Y,... because the average is so uncertain" and if so how to determine that threshold.