Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have 40 respondents, they answered on their preferred webpage after seeing page A and page B. Results say that 22 respondents preferred A and 18 preferred B:
Prefer A = 22
Prefer B = 18
n = 40
How can I test if there are differences among the A and B given the small sample size? Can I test the differences?
Certainly you can test for a difference of proportion (it's equivalent to testing $P(\text{prefers A}) = \frac12$). The usual test would be the binomial test, or you might use a normal approximation to the binomial (in which case, with $p_0=\frac12$ I'd suggest using a continuity correction), or a chi-squared goodness of fit test (or indeed other tests are possible).
However, on those data you will have no hope of rejecting the null at any reasonable significance level; under the null, the standard error of the difference between the counts will be larger than the observed difference; with $n=40$ you'd need an observed difference more than three times as big for a two-tailed p-value to reach typical significance levels.
