# Survey sampling: How many participants per group (current, former, and potential users) should we recruit?

We are designing research in which we will explore the perception of users (C current, F former, and P potential) on a product. We apply the same questionnaire for each group adapting (verb tenses) some questions depending on user group (C, F, and P). We use five-point Likert scales, multiple and single choices questions to measure intention to use the product, and willingness to give feedback, among others factors. We adapted some scales from previous studies like Lajunen and Summala [1] (attitude towards driving).

We want to compare the results from each group to conclude, for example: current users are more willing to get feedback using the product than potential users, former users are more willing to share their personal information using this product than potential users, etc.

After running a screening survey we identified 30 current, 70 former, and 400 potential users of the product that we want to evaluate. Thus, we could recruit a maximum of 30 current users and 70 former users (the best-case scenario).

We want to know if recruiting different sample sizes for the three groups (C current, F former, and P potential users) could add value to our analysis, for example running the survey with 30C/70F/70P, or 30C/70F/200P. In case of deciding to conduct the study with different sample sizes for each group:

-> How many participants per group should we recruit?

-> How could we analyse/compare the output from a small sample (less than 30) with another big sample (more than 100)?

-> Could we compare the analysis from a parametric test with a non-parametric test?

[1] Lajunen, T., & Summala, H. (1995). Driving experience, personality, and skill and safety-motive dimensions in drivers' self-assessments. Personality and Individual Differences, 19(3), 307-318.

We also appreciate getting any relevant literature.

In terms of questionnaire results, you give no idea how you will assess willingness of subjects to give feedback. I guess, some kind of questionnaire score.

The required sample size $$n$$ depends on what test you will use, the significance level of the contemplated test test (5%?), the size $$\Delta$$ of score difference you want to detect, the desired probability of detecting those differences, and the common standard deviation $$\sigma$$ of the scores at the three levels of the factor.

So more input is required in order to determine the required sample size.

As a general rule it would be a more efficient use of subjects to have equal sample sizes in each of the three groups. You ask what sample size to use, and yet you say you will have sizes 20, 70, & 400. How did you get those sample sizes? What are you really asking? Just for "statistical blessing" of what you've already decided to do?

Below are results from Minitab's power and sample size procedure for a one-factor ANOVA. Scores are assumed to be normal with standard deviation $$\sigma = 15.$$ For an ANOVA test at the 5% level, seeking to detect differences of sizes $$\Delta = 5, 10.$$ and power levels 80% and 90% (probabilities of detecting differences of sizes $$\Delta)$$ the output shows the number required in each group.

I don't suppose your data and objectives will match my assumptions, but this is one way to illustrate the kind of information that is required to answer your question.

Notes:

• Most of these pre-programmed power and sample size procedures assume that all three levels of the factor will have have the same number of subjects. If you insist on different sample sizes, other methods can be used.

• If your data are not normal, then you would not use ANOVA to analyze your data, and other methods of finding required sample sizes would be required.]

Minitab Output:

Power and Sample Size

One-way ANOVA

α = 0.05  Assumed standard deviation = 15

Factors: 1  Number of levels: 3

Maximum  Sample  Target
Difference    Size   Power  Actual Power
5     175     0.8      0.801385
5     229     0.9      0.900303
10      45     0.8      0.806033
10      58     0.9      0.900245

The sample size is for each level.

Power Curve for One-way ANOVA


Other quantities being equal, it is the ratios of $$\Delta/\sigma$$ that matter. So we would get the same recommended sample sizes for $$\Delta = 1, 2$$ with $$\sigma = 3.$$

The power curve below, gives an idea of power values for different $$\Delta$$s.

Addendum per comments: Rough power values for K-W test on normal data.

Sample sizes 30, 70, 70, with $$\sigma=15,$$ differences in means as shown, power about 78% for detecting largest difference $$\Delta=10.$$

set.seed(2021)
pv = replicate(10^4, kruskal.test(list( rnorm(30,100,15),
rnorm(70,105,15), rnorm(70,110,15) ))$p.val) mean(pv <= .05) [1] 0.7811  Change to sample sizes 30, 70, 200, gives power about 92%. set.seed(2021) pv = replicate(10^4, kruskal.test(list( rnorm(30,100,15), rnorm(70,105,15), rnorm(200,110,15) ))$p.val)
mean(pv <= .05)
[1] 0.9254


This is good news if it's the largest difference that really matters.

• I think the intention of the question is to understand whether given the sample size of 30 current (C) users (more of these are hard to recruit), does it makes sense to recruit more participants for the other categories (F former, P potential)? In other words, is there added value in going from 30C/30F/30P to 30C/70F/400P, or 30C/70F/70P? In the first case the sample sizes are equal, but they're equally small; in the latter cases some categories have more participants, but the sample size is not the same. Commented Jun 30, 2021 at 9:37
• Thanks for the reply @BruceET. I edited the question to provide additional context and more details about the problem. Have I answered your questions? Commented Jul 2, 2021 at 6:06
• @ralien you are right. This is the intention of the question. We want to know if recruiting different sample sizes for the three groups (C current, F former, and P potential users) could add value to our analysis, for example running the survey with 30C/70F/70P, or 30C/70F/200P. Commented Jul 2, 2021 at 6:07
• Going from 30C/70F/70P, to 30C/70F/200P would increase the probability of finding a difference between P and the other two groups, but not much if any to the probability of distinguishing between C and F. // I could do a simulation to show the approx effect. Summaries of some of your Likert data might make the simulation results more realistic. Are you planning to use Kruskal-Wallis test to compare the three groups? (Simulation results w/ ANOVA might give rough clues what to expect from K-W.) What difference in Likert medians is important to detect> .// Unclear why you deleted your question. Commented Jul 2, 2021 at 8:10
• See Addendum to answer. Commented Jul 2, 2021 at 8:51