# Analysis of data collected from the same sample, but in two experiments

My hypothesis is that users who used application X will have lower satisfaction levels than users who used application Y. My collected data-set looks like this:

+-------------+-----------------+-----------------+
|             | Satisfaction(X) | Satisfaction(Y) |
+-------------+-----------------+-----------------+
| Satisfied   | 9               | 14              |
+-------------+-----------------+-----------------+
| Neutral     | 4               | 0               |
+-------------+-----------------+-----------------+
| Unsatisfied | 2               | 1               |
+-------------+-----------------+-----------------+
| Total       | 15              | 15              |
+-------------+-----------------+-----------------+


These numbers represent the frequency counts, for example 9 people answered that they are satisfied with application X, 2 were Neutral ect... These two groups were identical, they tested application X and then tested application Y. What type of statistical analysis is best suited for confirmation/rejection of my hypothesis?

## 1 Answer

Assuming that of 30 subjects, 15 were randomly assigned to try application X and the other 15 to try application Y, a chi-squared test of homogeneity does not reject the null hypothesis that that the two applications are deemed equally satisfactory. On the first try, the test warns of too little data for the so-called chi-squared statistic to have an approximately chi-squared distribution.

chisq.test(TBL)

Pearson's Chi-squared test

data:  TBL
X-squared = 3.862, df = 2, p-value = 0.145

Warning message:
In chisq.test(TBL) : Chi-squared approximation may be incorrect


However, the implementation of chisq.test in R can simulate an approximate P-value, which also fails to reject the null hypothesis.

chisq.test(TBL, sim=T)

Pearson's Chi-squared test
with simulated p-value
(based on 2000 replicates)

data:  TBL
X-squared = 3.862, df = NA, p-value = 0.1909


However, the chi-squared test is inherently two-sided, and your question indicated you want to test whether X is more favorably received. One way to do a one-sided test would be to consider only favorable votes: X got 9 favorable votes out of 15, and Y got 14 favorable votes out of 15.

So we can do a one-sided test of proportions using prop.test in R. This test gives a P-value above 5%, indicating no significant difference between X and Y. [Again, there is a warning message about too little data.]

prop.test(c(9,14), c(15,15))

2-sample test for equality of proportions
with continuity correction

data:  c(9, 14) out of c(15, 15)
X-squared = 2.9814, df = 1, p-value = 0.08423
alternative hypothesis: two.sided
95 percent confidence interval:
-0.6782055  0.0115388
sample estimates:
prop 1    prop 2
0.6000000 0.9333333

Warning message:
In prop.test(c(9, 14), c(15, 15)) :
Chi-squared approximation may be incorrect


Although your small experiment with 30 gives some indication that Y may be better than X, it does not have enough information to say for sure.