I am trying to analyze a poll result for two different message treatments (T1 and T2) that essentially asks "Was this message helpful?" and there are two responses, "Yes", and "No" (but should be noted that not all survey get a response). We want to see which message tests better with subjects.

Someone suggested using a t-test to compare the number of positive responses. But I'm confused as to how to apply the t-test given that the "mean" of all the positive responses is always 1. The difference between the two populations would be the number of positive responses.

One thought I had was to say a "No" was equal to 0 and and "Yes" was equal to 1 and then I could calculate a mean of the two populations. But then what wouldn't be clear is what to do with surveys that didn't get a response

Ultimately I'm trying to understand if the T1 and T2 messages yield statistically different distributions to the poll

  • $\begingroup$ What questions do you have about the poll responses? $\endgroup$
    – Dave
    Commented Dec 1, 2020 at 2:00
  • $\begingroup$ I'm trying to understand if the two message treatments yield statistically different distributions of yes's/no's/no responses. I've expanded upon this a bit in my question $\endgroup$
    – sedavidw
    Commented Dec 1, 2020 at 2:50
  • $\begingroup$ Check out the chi-squared test or Fisher’s exact test. You’re essentially checking if two dice are weighted the same. $\endgroup$
    – Dave
    Commented Dec 1, 2020 at 2:55
  • $\begingroup$ Wouldn't that require that I know the expected distribution of both populations and would compare if the observed distribution matched the expected? $\endgroup$
    – sedavidw
    Commented Dec 1, 2020 at 3:08
  • $\begingroup$ You can treat it like comparing if two dice have the same weighting (not if two dice are fair, which is a related but different problem). Chi-squared and Fisher tests are appropriate here. $\endgroup$
    – Dave
    Commented Dec 1, 2020 at 3:25

1 Answer 1


If you have 210 Yes's and 183 No's for T1 and 382 Yes's and 205 No's for T2, then you could say that's 210 Yes's out of 393 responses for T1 and 382 Yes's out of 587 responses for T2. In R, a prop.test can test whether the rate of Yes's out of responses is the same for T1 and T2 (for details of a similar test see this page.):

prop.test(c(210,382), c(293,587), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(210, 382) out of c(293, 587)
X-squared = 3.8618, df = 1, p-value = 0.0494
alternative hypothesis: two.sided
95 percent confidence interval:
 0.00154269 0.13037119
sample estimates:
   prop 1    prop 2 
0.7167235 0.6507666 

(Because of the large sample sizes, I suppressed continuity correction.) The difference between T1 and T2 is barely significant at the 5% level.

If you worry that non-responses might be 'polite' No's (or otherwise of interest), then you could compare numbers of Yes's out of total surveys in each group (prop.test not shown).

If you want to know if results Yes, No, and NR are homogeneous across T1 and T2, then you could do a chi-squared test on a $2 \times 3$ table, as follows:

t1 = c(210, 183, 47);  t2 = c(382, 205, 60)
TBL = rbind(t1, t2);  TBL
   [,1] [,2] [,3]
t1  210  183   47
t2  382  205   60

        Pearson's Chi-squared test

data:  TBL
X-squared = 13.884, df = 2, p-value = 0.0009664

        [,1]      [,2]       [,3]
t1 -1.914206  2.070177  0.5604064
t2  1.578567 -1.707190 -0.4621439

The null hypothesis of homogeneity is strongly rejected.

The chi-squared statistic $Q = 13.885$ is the sum of six 'contributions' $C_{ij} = \frac{(X_{ij}-E_{ij})^2}{E_{ij}},$ where $X_{ij}$ are observed counts from TBLand $E_{ij}$ are computed from row and column totals of TBL using the null hypothesis. Residuals are square roots of $C_{ij}$ with signs depending on the sign of the numerator before squaring. The residuals with largest absolute values may point the way to useful ad hoc tests.

While the non-responses contribute to the overall significance (in my hypothetical data), they do not seem to be candidates for ad hoc tests.

Following, @Dave's Comments, you could use chi-squared tests or Fisher exact tests on $2 \times 2$ tables, instead of either prop.test.

  • $\begingroup$ 1. On the first link, what are the six syntaxes? They represent a generic language, not R? 2. The extra information from the chi-squared test is helpful to answer the question, as it uses all the data. If I was reporting these results I would be tempted to put that first and then the proportion test below if that was of particular a priori interest. Does using both entail multiplicity issues, not only are the tests using the same data their significance results are going to be “correlated” in a sense. $\endgroup$ Commented Dec 1, 2020 at 10:24
  • $\begingroup$ The NIST link shows (at least approximately) what prop.test does. // For my hypothetical (simulated) data, prop.test barely rejects at the 5% level, so you might not want to use prop.test as an ad hoc follow-up to results of chisq.test, which rejects at a smaller level of significance. Your concern about possible false discovery is on target. However, this is a question that should be answered for your actual data. My simulated data are introduced so I can show R syntax and output, not to recommend an analysis plan for your data. $\endgroup$
    – BruceET
    Commented Dec 2, 2020 at 0:59
  • $\begingroup$ 1. I realised from your resonse and the link that you were referencing this for the formulae, rather than the syntax below this and the formulae are helpful to see what is approximately done. 2. Apologies for this, I was reading the hypothetical values as occurred variates! Nonetheless, these values and R result output helped to think about the question. I would still be inclined to report as per my previous comment, this having been decided at the outset. Thanks for clarification, regarding false discovery. $\endgroup$ Commented Dec 2, 2020 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.