Suppose every person in a group ranks the quality of two products. The quality can be ranked as 1 (very good), 2, 3, 4 and 5 (very bad). I have a list of those observations. which is the right way for me to compare if the quality of products A and B has the same distribution? And how can I implement it in Stata?
1 Answer
This seems to be a paired design: each subject tests both Product A and Product B. Suppose you have $n = 70$ subjects. Also, suppose that B is somewhat worse than A, to the extent that occasional subjects give B a slightly higher (less favorable) score, while very occasional subjects give A a slightly higher score.
Fake data for our experimentation here might be generated as follows:
set.seed(1887); m = 70
A = sample(1:5, m, rep=T, p=c(2,2,1,1,0))
d = sample(-1:1, m, rep=T, p=c(1,4,2)) # 1/7 like B better; 2/7 like A better
B = pmin(A+d, 5)
summary(A); summary(B); summary(B-A)
Min. 1st Qu. Median Mean 3rd Qu. Max. ## A
1.000 1.000 2.000 1.943 2.000 4.000
Min. 1st Qu. Median Mean 3rd Qu. Max. ## B
0.0 1.0 2.0 2.1 3.0 4.0
Min. 1st Qu. Median Mean 3rd Qu. Max. ## B-A
-1.0000 0.0000 0.0000 0.1571 1.0000 1.0000
A Wilcoxon signed rank test on differences B - A
would not work well because
of the preponderance of ties and 0's. Also, there are very few different integer values in B - A
so that it seems too much of a stretch to assume normality and do a t test.
A tally of B-A is as follows:
table(B-A)
-1 0 1
8 43 19
That is, among those who indicated any difference between A and B, 19 out of 27 preferred A.
A 'sign test' or 'binomial test' assesses whether equally preferred products A and B would show such a distribution of lower and higher scores, the null hypothesis. In R, the procedure binom.test
, with a 2-sided alternative, gives the following results:
binom.test(19, 27)
Exact binomial test
data: 19 and 27
number of successes = 19, number of trials = 27, p-value = 0.05224
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
0.4981863 0.8624734
sample estimates:
probability of success
0.7037037
The P-value slightly greater than 5% shows that we do not have evidence that
the products differ in favorability at the 5% level. In R, the P-value is computed from
the binomial CDF pbinom
as shown below:
2*(1-pbinom(18, 27, .5))
[1] 0.05223899
However, suppose B is a cheapened version of A and you anticipated that B would be rated worse than A, if anything. Then a one-sided test would be appropriate, as shown below. It shows significance at the 5% level.
binom.test(19, 27, alte="g")
Exact binomial test
data: 19 and 27
number of successes = 19, number of trials = 27, p-value = 0.02612
alternative hypothesis: true probability of success is greater than 0.5
95 percent confidence interval:
0.5286085 1.0000000
sample estimates:
probability of success
0.7037037
Notes: (a) For this illustration I have chosen fake data that shows relatively little difference between products. If there were a more substantial difference, then 70 subjects would likely show a significant difference in the two-sided test. (b) This page
shows documentation for the R procedure SIGN.test
(not in base R), which would
accept the vector B-A
as input [instead of counts derived from this vector, as for binom.test
.] (c) It has been some time since I used Stata; the first thing to check is whether it implements a 'sign test' to your satisfaction. For promising links, google sign test Stata
. Otherwise, you can use its binomial PDF function to find P-values.
Addendum: As I mentioned earlier, the Wilcoxon signed-rank test is of questionable value here because there are so few unique values in the difference B-A. In my example, three: $-1, 0, 1,$ mostly $0$'s, so that that nothing is lost using the sign test. In general with differences in Likert scores, there can be very few unique non-zero values unless differences of opinion about A and B are profound.
length(unique(B-A))
[1] 3
wilcox.test(B-A)
Wilcoxon signed rank test with continuity correction
data: B - A
V = 266, p-value = 0.03545
alternative hypothesis: true location is not equal to 0
Warning messages:
1: In wilcox.test.default(B - A) : cannot compute exact p-value with ties
2: In wilcox.test.default(B - A) :
cannot compute exact p-value with zeroes
If there were only a few ties or 0's, maybe I'd be willing to trust the approximate value given, but this situation presents extreme cases. Then claims in a proprietary technical bulletin that all is well could hardly be persuasive.
Artificially breaking ties by slight jittering (as with wilcox.test(B-A+runif(70,-.1,.1))
) gave P-values from .03 to .46 in a dozen tries, only two
of them below 5%. I believe power is not being 'hemoraged'; it was never
there.
Perhaps a permutation test would have better power than a sign test, but there can't be many unique medians in a permutation distribution.
-
$\begingroup$ The sign test hemorrhages statistical power relative to the sign rank test because it discard information about the (ranked) magnitude of the differences. $\endgroup$– AlexisCommented Aug 7, 2018 at 0:17
-
$\begingroup$ True, but here there are too many ties and 0's for the signed rank test. $\endgroup$– BruceETCommented Aug 7, 2018 at 0:19
-
2$\begingroup$ Stata's signed rank test implementation adjusts for both zeros and ties out of the box. Article on it: Sribney, W. M. (1995). Correcting for ties and zeros in sign and rank tests. Stata Technical Bulletin, 26:2–4. $\endgroup$– AlexisCommented Aug 7, 2018 at 0:22
-
$\begingroup$ An exact Wilcoxon signed-rank test in the presence of ties is provided by the
exactRankTests
package for R. (For your example it should result in exactly the same p-value as the sign test.) Nevertheless, there are a couple of reasons one might prefer a sign test, though I think you've not quite put your finger on them. First, the null hypothesis for the Wilcoxon signed-rank test is that the ranked differences are symmetric around nought - but asymmetry needn't result from A's being better than B (or vice versa) in any sense. ... $\endgroup$– Scortchi ♦Commented Aug 7, 2018 at 12:57 -
$\begingroup$ ... Second, to rank differences you need to be able to take differences - getting around this by first transforming "very good" &c. into ranks isn't always what you'd want to do. The sign test doesn't require a decision about whether '1' for A & '3' for B is a difference of larger or smaller magnitude than '5' for A & '4' for B. $\endgroup$– Scortchi ♦Commented Aug 7, 2018 at 12:57
signrank
in Stata). In either case the null hypothesis and alternate hypothesis is the same: $H_{0}: P(X_{A} > X_{B}) = 0.5; H_{A}: P(X_{A} > X_{B}) \ne 0.5$ $\endgroup$