I have 40 respondents, they answered on their preferred webpage after seeing page A and page B. Results say that 22 respondents preferred A and 18 preferred B:

Prefer A = 22
Prefer B = 18
n = 40

How can I test if there are differences among the A and B given the small sample size? Can I test the differences?


1 Answer 1


Certainly you can test for a difference of proportion (it's equivalent to testing $P(\text{prefers A}) = \frac12$). The usual test would be the binomial test, or you might use a normal approximation to the binomial (in which case, with $p_0=\frac12$ I'd suggest using a continuity correction), or a chi-squared goodness of fit test (or indeed other tests are possible).

However, on those data you will have no hope of rejecting the null at any reasonable significance level; under the null, the standard error of the difference between the counts will be larger than the observed difference; with $n=40$ you'd need an observed difference more than three times as big for a two-tailed p-value to reach typical significance levels.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.