To restate things and make sure there's no misunderstanding:
You have a "reference dataset" (i.e. your "previously analyzed dataset") with a categorical distribution among 12 levels, maybe looking like that (in R):
#note that this vector is sorted by decreasing order, the maximum proportion being 30%, and the smallest 1%
reference_distribution = c(0.30, 0.15, 0.12, 0.08, 0.08, 0.07, 0.06, 0.04, 0.03, 0.03, 0.03, 0.01)
#But let's make sure this is sorted in decreasing order anyway - this will be important for coding purposes later.
reference_distribution = sort(reference_distribution, decreasing=TRUE)
You plan to collect data from another source (or have already collected the data), and want to compare this new sample to the reference distribution above, to see if their discrepancies (that will inevitably happen) are concerning. You want to make sure that the size of your new sample is large enough to get a reliable result relative to these differences, or if you need to collect additional data.
In short: when you need to calculate a sample size, you have to, in this order:
- determine the statistical test adequate for your purpose;
- determine the minimal effect size you're interested in;
- calculate the required sample size.
1- Choosing an appropriate test
An appropriate statistical test here to compare your new sample to the reference distribution is almost certainly a chi-square goodness-of-fit test, which is meant to compare categorical distributions. In short, the chi-squared test answers the question: "If the sample really comes from the reference distribution, how likely is it for this sample to look like this?".
Had your data being paired (that is, the same customers answered the first and second survey, and you wanted to check if these customers changed their mind between the first and second survey), a more appropriate test would have been the Stuart-Maxwell test. It would have implied a different (and probably more complicated) procedure for calculating the required sample size. Here, we'll stick to the chi-square goodness-of-fit test, as it seems more appropriate for your data.
You'll find a short documentation for the chi-squared goodness-of-fit test in R on this page. This is an extremely common statistical test, and many free online resources (including this very website) are available on how to interpret correctly the result of this test. So I don't think it is necessary to explain it in detail in this answer.
If the chi-squared test shows that something is going on (i.e. that there's an overall difference between your sample and the reference distribution), you might also want to examine the residuals to see which levels are driving this overall difference. You'll find a technical but useful discussion on residuals in this article by D. Sharpe (2015).
But this short presentation of the chi-squared test was just an aside, and using this test will come after you collected your data. The important step here was simply to identify in advance the test to use after the data has been collected. It is important to identify the correct test beforehand, because it affects the way to calculate the required sample size.
2- Defining a minimal effect size you care about
The ability of the chi-square goodness-of-fit test to detect a difference if it exists is called its power. As the sample size increases, power increases. A consequence of that is if the sample size is too small, you might miss a difference you're interested in, and this is probably what your stakeholders are worried about. But on the other hand, if the sample size is too large, the test may detect differences that actually do not matter to you, so it might make you worry when there's no reason to (a note on that at the end of this answer).
So you want to run an a priori power analysis for the chi-square goodness-of-fit test, which will give you an appropriate sample size to detect the minimal effect you're interested in.
In R, the pwr library may help you conducting this kind of analysis for the chi-square test. Other statistical software or languages generally also offer this kind of feature, e.g. GPower which is specialized in power analysis, or the statsmodels library in Python.
To conduct a power analysis, you have to define beforehand the magnitude of the difference you want to be able to detect between the sample and its hypothesized distribution.
For that, you can ask yourself what kind of difference it would be problematic or not to miss.
To take an example, let's say you're interested in detecting a deviation/difference of at least $\pm8\%$ in at least one level/cell (because for whatever reasons specific to your business, you know that deviations smaller than 8% would not really matter).
diff = 0.08 #this is the difference we want to be able to detect in at least one cell, to let's enter that in R.
"A deviation of 8% in at least one cell" is a very specific criteria, and in the following 3 or 4 paragraphs, I'm going to detail quite a bit how to deal with it. But you don't have to use this kind of criteria or to follow exactly the specific workflow I'm going to suggest. It's just an example to give you an idea of how to think about it. You may have better ideas of criteria relevant to your situation. The main idea here is that smaller the difference you care about, the bigger the sample size will have to be - so you have to think carefully about what kind of difference/deviation is of interest to you.
Returning to our example, one of the "worst case scenarios" would be if this difference of $\pm8\%$ happened in the largest proportion cell of the reference distribution (in our example above, 0.3), because differences in largest cells are more difficult to detect for the chi-square goodness-of-fit test. Here, it would mean a proportion of 0.22 for this level in the new sample (it could have been 0.38 too, but let's take the example of 0.22).
An "even worst case scenario" is if the 8% you'd have to add elsewhere are added to each remaining level in a proportional way (it makes the deviation even harder to detect).
Let's create a hypothetical sample to simulate this "worst case scenario":
new_sample = c( reference_distribution[1] - diff) #we substract 8% from 30%. For the moment, this vector is just c(0.22)
#let's add 8% proportionally to the remaining levels
remaining_prop = reference_distribution[2:length(reference_distribution)] #we retrieve a vector containing only the remaining levels
remaining_prop = remaining_prop + (remaining_prop/sum(remaining_prop)* diff) #we add the 8% proportionally to each remaining level
new_sample= c(new_sample, remaining_prop )
new_sample
[1] 0.22000000 0.16714286 0.13371429 0.08914286 0.08914286 0.07800000 0.06685714 0.04457143 0.03342857 0.03342857 0.03342857 0.01114286
So we now have a reference distribution and a hypothetical "worst-case scenario" sample, that look like that if we compare their distributions side by side:
| level |
Reference distribution |
Hypothetical "worst-case scenario" sample |
| A |
0.30 |
0.22 |
| B |
0.15 |
0.16714286 |
| C |
0.12 |
0.13371429 |
| D |
0.08 |
0.08914286 |
| E |
0.08 |
0.08914286 |
| F |
0.07 |
0.07800000 |
| G |
0.06 |
0.06685714 |
| H |
0.04 |
0.04457143 |
| I |
0.03 |
0.03342857 |
| J |
0.03 |
0.03342857 |
| K |
0.03 |
0.03342857 |
| L |
0.01 |
0.01114286 |
From this hypothetical "worst-case" sample and from the reference distribution, we are now able to calculate an effect size, essentially measuring the deviation of the sample from the reference distribution. We will then plug this effect size into the appropriate pwr method to get the required sample size.
The appropriate effect size for a chi-square goodness-of-fit test is called Cohen's $\omega$ (this is the Greek letter omega, but some people simply call it "Cohen's w").
Here's how to calculate the effect size Cohen's $\omega$ with the pwr library, using the reference distribution and the hypothetical sample we created before:
library(pwr)
effect_size_w = ES.w1(reference_distribution, new_sample)
effect_size_w
>>>0.1745743
Here, $\omega = 0.1745743$. (In case you want to calculate $\omega$ manually, you can find the formula on the Wikipedia article about effect sizes).
Higher $\omega$ is, more different the two distributions are, and lower it is, more similar they are. Here, 0.1745743 is the smallest possible $\omega$ we get given a criteria of "a difference of at least 8% in at least one cell". If we want to err on the side of caution, we can round it to 0.17:
effect_size_w = round(effect_size_w , 2)
Now that we have the smallest possible effect size we care about, we're almost ready to calculate the sample size required to detect an effect size of at least this magnitude.
3- Calculating the required sample size
We still have to define an alpha level, and the power of the test. Choosing 0.05 for alpha is very common, but it entirely depends on your use case and you may want to choose a smaller level (see this discussion). A power of 0.8 is also very common to choose (but here again, you may want a higher level).
So we just have to plug in pwr the effect size we calculated earlier ($\omega=0.17$), the alpha level, the power, and the degrees of freedom (number of levels minus one). With an alpha level of 0.01 and a power of 0.9, here is what we'd get:
pwr.chisq.test(w=effect_size_w, df=(12-1), sig.level=0.01, power=0.9)
Chi squared power calculation
w = 0.17
N = 961.857
df = 11
sig.level = 0.01
power = 0.9
NOTE: N is the number of observations
So this a required sample size (N) of 962 for this specific example.
You may want to adjust this number upward to account for possible problems like non-responses, duplicate answers, and other shenanigans, depending on your knowledge of the people you're surveying.
4- After collecting the data
We collected our data, and maybe we have a sample looking like that:
real_sample = c(209, 98, 79, 66, 115, 87, 58, 60, 77, 65, 22, 26)
#in proportions, this is c(0.21725572, 0.10187110, 0.08212058, 0.06860707, 0.11954262, 0.09043659, 0.06029106, 0.06237006, 0.08004158, 0.06756757, 0.02286902, 0.02702703)
If we compare our collected sample to the reference distribution, we have something like that:
| level |
Reference distribution |
real collected sample |
| A |
0.30 |
0.21725572 |
| B |
0.15 |
0.10187110 |
| C |
0.12 |
0.08212058 |
| D |
0.08 |
0.06860707 |
| E |
0.08 |
0.11954262 |
| F |
0.07 |
0.09043659 |
| G |
0.06 |
0.06029106 |
| H |
0.04 |
0.06237006 |
| I |
0.03 |
0.08004158 |
| J |
0.03 |
0.06756757 |
| K |
0.03 |
0.02286902 |
| L |
0.01 |
0.02702703 |
We can run the chi-square test to check how likely it is that this sample comes from the reference distribution:
chisq.test(real_sample, p=reference_distribution)
Chi-squared test for given probabilities
data: real_sample
X-squared = 2606.2, df = 11, p-value < 2.2e-16
Here, with a very small p-value (<2.2e-16), no way this sample comes from the reference distribution. So something is going on, the too distributions are too different to ignore it given the criteria we previously defined. Maybe the reference distribution was not reflecting correctly how the population is, maybe there were many categorization errors in the second dataset, or maybe something changed for customers between the time the first dataset and second dataset have been collected. There are many possible explanations that should be investigated, but anyway something is going on here.
Caveat about a too large sample size
If you already collected data before conducting sample size calculations, and you realize that you have really much more data than what you actually need, it may be a problem - but nothing unsolvable.
Indeed, in this case, the chi-square test may return a very small p-value even if the difference between the sample and the reference distribution is actually of no practical interest to you. If you're in this situation (sample size much larger than what you calculated) and are unsure on how to interpret the small p-value you get from your test, I'd suggest to have a look at the following pages: Sample size too large? and Are large data sets inappropriate for hypothesis testing?
Bibliography
You can find a thorough discussion of power calculations for the chi-squared tests in the chapter 7 of: Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Routledge.
If you want to use other tools than R, there are many online video tutorial explaining on to do that using Gpower. You'll also find a comparison on how to perform power calculations in R and Python here, section "Chi-square goodness-of-fit": Perktold, J. (2013, March 17). joepy: Statistical Power in Statsmodels. Joepy. http://jpktd.blogspot.com/2013/03/statistical-power-in-statsmodels.html
