Survey sampling: How many participants per group (current, former, and potential users) should we recruit?

Question

We are designing research in which we will explore the perception of users (C current, F former, and P potential) on a product. We apply the same questionnaire for each group adapting (verb tenses) some questions depending on user group (C, F, and P). We use five-point Likert scales, multiple and single choices questions to measure intention to use the product, and willingness to give feedback, among others factors. We adapted some scales from previous studies like Lajunen and Summala [1] (attitude towards driving).

We want to compare the results from each group to conclude, for example: current users are more willing to get feedback using the product than potential users, former users are more willing to share their personal information using this product than potential users, etc.

After running a screening survey we identified 30 current, 70 former, and 400 potential users of the product that we want to evaluate. Thus, we could recruit a maximum of 30 current users and 70 former users (the best-case scenario).

We want to know if recruiting different sample sizes for the three groups (C current, F former, and P potential users) could add value to our analysis, for example running the survey with 30C/70F/70P, or 30C/70F/200P. In case of deciding to conduct the study with different sample sizes for each group:

-> How many participants per group should we recruit?

-> How could we analyse/compare the output from a small sample (less than 30) with another big sample (more than 100)?

-> Could we compare the analysis from a parametric test with a non-parametric test?

[1] Lajunen, T., & Summala, H. (1995). Driving experience, personality, and skill and safety-motive dimensions in drivers' self-assessments. Personality and Individual Differences, 19(3), 307-318.

We also appreciate getting any relevant literature.

Answer 1

In terms of questionnaire results, you give no idea how you will assess willingness of subjects to give feedback. I guess, some kind of questionnaire score.

The required sample size $n$ depends on what test you will use, the significance level of the contemplated test test (5%?), the size $\Delta$ of score difference you want to detect, the desired probability of detecting those differences, and the common standard deviation $\sigma$ of the scores at the three levels of the factor.

So more input is required in order to determine the required sample size.

As a general rule it would be a more efficient use of subjects to have equal sample sizes in each of the three groups. You ask what sample size to use, and yet you say you will have sizes 20, 70, & 400. How did you get those sample sizes? What are you really asking? Just for "statistical blessing" of what you've already decided to do?

Below are results from Minitab's power and sample size procedure for a one-factor ANOVA. Scores are assumed to be normal with standard deviation $\sigma = 15.$ For an ANOVA test at the 5% level, seeking to detect differences of sizes $\Delta = 5, 10.$ and power levels 80% and 90% (probabilities of detecting differences of sizes $\Delta)$ the output shows the number required in each group.

I don't suppose your data and objectives will match my assumptions, but this is one way to illustrate the kind of information that is required to answer your question.

Notes:

• Most of these pre-programmed power and sample size procedures assume that all three levels of the factor will have have the same number of subjects. If you insist on different sample sizes, other methods can be used.

• If your data are not normal, then you would not use ANOVA to analyze your data, and other methods of finding required sample sizes would be required.]

Minitab Output:

Power and Sample Size 

One-way ANOVA

α = 0.05  Assumed standard deviation = 15

Factors: 1  Number of levels: 3


   Maximum  Sample  Target
Difference    Size   Power  Actual Power
         5     175     0.8      0.801385
         5     229     0.9      0.900303
        10      45     0.8      0.806033
        10      58     0.9      0.900245

The sample size is for each level.


 Power Curve for One-way ANOVA 

Other quantities being equal, it is the ratios of $\Delta/\sigma$ that matter. So we would get the same recommended sample sizes for $\Delta = 1, 2$ with $\sigma = 3.$

The power curve below, gives an idea of power values for different $\Delta$s.

Addendum per comments: Rough power values for K-W test on normal data.

Sample sizes 30, 70, 70, with $\sigma=15,$ differences in means as shown, power about 78% for detecting largest difference $\Delta=10.$

set.seed(2021)
pv = replicate(10^4, kruskal.test(list( rnorm(30,100,15), 
              rnorm(70,105,15), rnorm(70,110,15) ))$p.val)
mean(pv <= .05)
[1] 0.7811

Change to sample sizes 30, 70, 200, gives power about 92%.

set.seed(2021)
pv = replicate(10^4, kruskal.test(list( rnorm(30,100,15), 
              rnorm(70,105,15), rnorm(200,110,15) ))$p.val)
mean(pv <= .05)
[1] 0.9254

This is good news if it's the largest difference that really matters.