How to detect significant differences in the proportions of categorical variables? I am working on a project related to the way in which a message to smokers is communicated to reduce their smoking. For my experiment I will give a questionnaire to smokers where, in each question, they will be asked what message motivates them most to quit smoking. Option 1 is a classic message, option 2 is a message designed according to my theoretical framework, and the third option is a neutral option in that both options motivate them equally.
I would like to know, how can I evaluate if the differences, proportionally speaking, are significant? for example, if the total results of the proportions were:
classic message ----> 20%

message based on framework --->70%

neutral option message ---> 10%

How can I verify that these results are significant? Do I need to do a test for this?
Or can I simply justify it with the margin of error? The margin of error of my sample is 5%. Therefore the distribution at the population level in the worst case would be 65% of the message based on my framework, which would be much higher than the other two options, even if they are in the upper range of the margin of error. Would this analysis be sufficient?
Thanks for your help.
 A: Whenever you deal with proportions of categorical variables, you have
to use counts, not percentages.
Small sample. If you have $n = 10$ subjects, then you have observed counts X $(2, 7, 1)$ and expected count
$E = 10/3$ under the null hypothesis that all categories are equally likely to be chosen. Then you reject the null hypothesis for sufficiently large values of
the statistic $$Q = \sum_{i=1}^3 \frac{(X_i - E)^2}{E} \stackrel{aprx}{\sim}
\mathsf{Chisq}(\nu=3-1=2),$$ where the approximation is useful provide that $E \ge 5.$
n = 10;  p = c(.2,.7,.1);  X = n*p;  X
[1] 2 7 1
E = n/3
Q = sum((X-E)^2/E);  Q
[1] 6.2
c = qchisq(.99, 2);  c
[1] 9.21034
1 - pchisq(Q, 2)
[1] 0.0450492

Formally, one might reject $H_0$ at the 1% level of significance if $Q \ge c.$ Here, this is
is not true because the 'critical value` is $c = 9.210$ and $Q = 6.2.$
Also, the P-value $0.045$ (probability of a more extreme result, assuming $H_0$ is true)
is greater than $0.01 = 1\%.$
The difficulty is that $E = 3.33 < 5$ so that $Q$ may not have the distribution $\mathsf{Chisq}(\nu=2).$
In such circumstances, the procedure chisq.test in R is able to simulate
the actual distribution of $Q$ to get a useful approximate P-value. (See the simulation at the end of this Answer.)
Larger sample. However if you have $n = 100$ subjects with the same proportions, then you have
a very hightly significant result, as shown below:
n = 100;  p = c(.2,.7,.1);  X = n*p
E = n/3
Q = sum((X-E)^2/E);  Q
[1] 62
1 - pchisq(Q, 2)
[1] 3.441691e-14

Now, $Q = 62$ and the probability of a more extreme (i.e., larger) result
if the null hypothesis were true is very close to $0.$ (The critical value
for a test at the 1% level is still $c = 9.210.$ The critical value depends
on the number of categories (here three), not on the number of subjects.
Simulation in R for small sample. Here we use chisq.test in R to
get a useful P-value for based on a simulation of the actual distribution
of the so-called chi-squared statistic $Q.$ This test requires a matrix
with observed and expected counts for each of the three categories (answers to the survey question).
n = 10;  p = c(.2, .7, .1);  X = n*p
E = rep(10/3, 3)
MAT = rbind(X, E);  MAT
      [,1]     [,2]     [,3]
X 2.000000 7.000000 1.000000
E 3.333333 3.333333 3.333333
chisq.test(MAT, sim=T)

        Pearson's Chi-squared test 
        with simulated p-value 
        (based on 2000 replicates)

data:  MAT
X-squared = 2.8908, df = NA, p-value = 0.2734

So the outcome is that there is no statistically significant
difference among answers in the ten-subject experiment---not even at the 5% level.
A: I think you were on the right track when thinking about margin of error. But don't use the margin of error predicted by your sample size. Calculate the actual 95% (or some other level) confidence interval from the data you collect. You'll find plenty of web calculators that can compute the confidence interval of a proportion.
I suggest you avoid the whole concept of your results being "significant". That word is ambiguous and widely misunderstood and abused.
