Applying a t-test to a polling a result

Question

I am trying to analyze a poll result for two different message treatments (T1 and T2) that essentially asks "Was this message helpful?" and there are two responses, "Yes", and "No" (but should be noted that not all survey get a response). We want to see which message tests better with subjects.

Someone suggested using a t-test to compare the number of positive responses. But I'm confused as to how to apply the t-test given that the "mean" of all the positive responses is always 1. The difference between the two populations would be the number of positive responses.

One thought I had was to say a "No" was equal to 0 and and "Yes" was equal to 1 and then I could calculate a mean of the two populations. But then what wouldn't be clear is what to do with surveys that didn't get a response

Ultimately I'm trying to understand if the T1 and T2 messages yield statistically different distributions to the poll

Answer 1

If you have 210 Yes's and 183 No's for T1 and 382 Yes's and 205 No's for T2, then you could say that's 210 Yes's out of 393 responses for T1 and 382 Yes's out of 587 responses for T2. In R, a prop.test can test whether the rate of Yes's out of responses is the same for T1 and T2 (for details of a similar test see this page.):

prop.test(c(210,382), c(293,587), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(210, 382) out of c(293, 587)
X-squared = 3.8618, df = 1, p-value = 0.0494
alternative hypothesis: two.sided
95 percent confidence interval:
 0.00154269 0.13037119
sample estimates:
   prop 1    prop 2 
0.7167235 0.6507666 

(Because of the large sample sizes, I suppressed continuity correction.) The difference between T1 and T2 is barely significant at the 5% level.

If you worry that non-responses might be 'polite' No's (or otherwise of interest), then you could compare numbers of Yes's out of total surveys in each group (prop.test not shown).

If you want to know if results Yes, No, and NR are homogeneous across T1 and T2, then you could do a chi-squared test on a $2 \times 3$ table, as follows:

t1 = c(210, 183, 47);  t2 = c(382, 205, 60)
TBL = rbind(t1, t2);  TBL
   [,1] [,2] [,3]
t1  210  183   47
t2  382  205   60
chisq.test(TBL)

        Pearson's Chi-squared test

data:  TBL
X-squared = 13.884, df = 2, p-value = 0.0009664

chisq.test(TBL)$resi
        [,1]      [,2]       [,3]
t1 -1.914206  2.070177  0.5604064
t2  1.578567 -1.707190 -0.4621439

The null hypothesis of homogeneity is strongly rejected.

The chi-squared statistic $Q = 13.885$ is the sum of six 'contributions' $C_{ij} = \frac{(X_{ij}-E_{ij})^2}{E_{ij}},$ where $X_{ij}$ are observed counts from TBLand $E_{ij}$ are computed from row and column totals of TBL using the null hypothesis. Residuals are square roots of $C_{ij}$ with signs depending on the sign of the numerator before squaring. The residuals with largest absolute values may point the way to useful ad hoc tests.

While the non-responses contribute to the overall significance (in my hypothetical data), they do not seem to be candidates for ad hoc tests.

Following, @Dave's Comments, you could use chi-squared tests or Fisher exact tests on $2 \times 2$ tables, instead of either prop.test.