Which statistic test should I use in this situation?

Question

There are 10 questions in which we are comparing peoples' opinion about two conditions, "X"and "Y". Every question was answered by about 90 people who were different than other questions' respondents. The results are something like this:

    X  Y

Q1: 21, 47

Q2: 84, 10

...

Q10: 43, 38

I want to compare these two conditions. One of them is not normally distributed. Which statistic test should I use? Two-Sample t-test or the Paired t-test, Wilcoxon-Mann-Whitney, or something else?

Answer 1

I suppose you want to test whether there is significant evidence that either opinion X or Y has stronger support.

Simulated data. One model for your data, in which opinion X is very slightly more popular than opinion Y, might be that you have $N \sim \mathsf{Pois}(\lambda = 90)$ people answering each of the questions, and then that opinion X gets $x \sim \mathsf{Binom}(N, p=.55)$ votes and opinion Y gets $y = N-x$ votes.

Simulating this model, we have data:

set.seed(2018); N = rpois(10, 90)
x = rbinom(10, N, .55); y = N-x
cbind(x,y, d=x-y)
       x  y  d 
 [1,] 51 34 17
 [2,] 36 45 -9
 [3,] 51 41 10
 [4,] 53 53  0
 [5,] 51 36 15
 [6,] 56 38 18
 [7,] 59 34 25
 [8,] 50 41  9
 [9,] 49 49  0
[10,] 47 44  3

Below we show results of various tests to see which ones detect the slight difference in preference of opinion X over opinion Y assumed in the simulation of these data. All alternatives are two-sided.

Sign test. The simplest test is a sign test, which looks at the number of negative signs in the last column (disregarding the 0's). It does not reject (5% level) the null hypothesis that the two opinions are equally popular. The p-value is 0.0703. The sign test has relative low power. It "ignores" that there are several large positive values in the last column and only one negative value.

2*(pbinom(1, 8, .5))
[1] 0.0703125

Wilcoxon Signed Rank test. A (one-sample) Wilcoxon signed-rank test rejects with approximate p-value 0.035. The warning messages are because of the 0s and because $\pm 9$ is considered a tie.

wilcox.test(x-y)

        Wilcoxon signed rank test with continuity correction

data:  x - y
V = 33.5, p-value = 0.03547
alternative hypothesis: true location is not equal to 0

Warning messages:
1: In wilcox.test.default(x - y) : cannot compute exact p-value with ties
2: In wilcox.test.default(x - y) :
  cannot compute exact p-value with zeroes

One-sample t test. A one-sample t test rejects the null hypothesis. The p-value is 0.025. The data may not be normal, but the t test is known to be robust.

t.test(x-y)

        One Sample t-test

data:  x - y
t = 2.695, df = 9, p-value = 0.02459
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
  1.413365 16.186635
sample estimates:
mean of x 
      8.8 

Permutation test. A permutation test, using the sum of differences as metric, rejects the null hypothesis. When signs were randomly permuted, we observed 85 unique sums of differences. The p value is 0.015; it is the probability that sums of permuted differences lie outside the two vertical bars (at $\pm 88)$ in the histogram. The p-value is simulated and subject to unimportant variation with 100,000 iterations. Three subsequent runs gave essentially the same p-value. [The Wilcoxon test is a kind of permutation test, but the current test needs no adjustments for 0's and ties.]

d.obs = sum(x-y);  d.obs
[1] 88
d.perm = replicate( 10^5, sum((x-y)*sample(c(-1,1), 10, rep=T)) )
mean(abs(d.perm) > abs(d.obs))
[1] 0.01529 
length(unique(d.perm))
[1] 85

Summary. In summary, only the notoriously low-powered sign test failed to find a significant difference in popularity between opinions X and Y. The permutation test gave the smallest p-value. On various grounds, one might wonder whether the assumptions for the Wilcoxon and t tests are met.

Note: You could also do a chi-squared test of homogeneity using the matrix of x and y values. It gives a non-significant result, which indicates there is no significant difference in the distribution of opinions between X and Y across the ten questions. This test is not appropriate for assessing whether opinions X and Y are equally popular.

chisq.test(cbind(x,y))

        Pearson's Chi-squared test

data:  cbind(x, y)
X-squared = 10.894, df = 9, p-value = 0.2831

Answer 2

Since you have a table of counts, you will want to use methods appropriate for count data. These might include chi-square test and logistic regression.

The simplest approach for your data would be to treat each Question separately. You will be assessing if the proportion of X and Y follow a null 50% / 50% distribution. For the hypothesis test, you might perform a chi-square goodness-of-fit test or a binomial test. For the effect size, you might just present the proportions of X and Y, or you could use another statistic like odds ratio. You might want to construct confidence intervals for the proportions and present these as well. (Be sure to use a method that is appropriate for proportions).

A different question would be to ask if the distribution of X or Y changes across questions. Here you could use a chi-square test of association on the whole table (Questions on one axis, and X-Y on the other, as you laid it out in the question).

Another approach to this second question is to use logistic regression where X-or-Y is your dependent variable and Question is your independent variable. This approach is helpful if you have additional factors you want the model to consider. For example, if you had also recorded the sex of respondents and wanted to include that as a predictor.

For example, if we look at Question 1, we could conduct a binomial test.

binom.test(21, (21+47))

   # Exact binomial test
   # 
   # data:  21 and (21 + 47)
   # number of successes = 21, number of trials = 68, p-value = 0.002186
   # alternative hypothesis: true probability of success is not equal to 0.5
   # 95 percent confidence interval:
   # 0.2023635 0.4325608
   # sample estimates:
   # probability of success 
   #          0.3088235 

Here, the proportion of X being chosen is 0.31. This is significantly different than a null proportion of 0.50 (p = 0.002).

We could also report the 95% confidence interval of this proportion (0.20 to 0.43). This is helpful in presenting results in tabular or graphical form.