3
$\begingroup$

Suppose every person in a group ranks the quality of two products. The quality can be ranked as 1 (very good), 2, 3, 4 and 5 (very bad). I have a list of those observations. which is the right way for me to compare if the quality of products A and B has the same distribution? And how can I implement it in Stata?

$\endgroup$
3
  • 3
    $\begingroup$ Mann-Whitney test. $\endgroup$
    – stans
    Commented Aug 6, 2018 at 18:16
  • $\begingroup$ I do not see how a Mann-Whitney-Wilcoxon rank sum test would apply to this paired design. Also, as mentioned in my Answer, a Wilcoxon signed rank test seems problematic because Likert data will show many ties. $\endgroup$
    – BruceET
    Commented Aug 6, 2018 at 23:30
  • 1
    $\begingroup$ Ah... if these are paired observations (i.e. the same subject gets both A and B), then you would use the sign rank test (signrank in Stata). In either case the null hypothesis and alternate hypothesis is the same: $H_{0}: P(X_{A} > X_{B}) = 0.5; H_{A}: P(X_{A} > X_{B}) \ne 0.5$ $\endgroup$
    – Alexis
    Commented Aug 7, 2018 at 0:15

1 Answer 1

-2
$\begingroup$

This seems to be a paired design: each subject tests both Product A and Product B. Suppose you have $n = 70$ subjects. Also, suppose that B is somewhat worse than A, to the extent that occasional subjects give B a slightly higher (less favorable) score, while very occasional subjects give A a slightly higher score.

Fake data for our experimentation here might be generated as follows:

set.seed(1887); m = 70
A = sample(1:5, m, rep=T, p=c(2,2,1,1,0))
d = sample(-1:1, m, rep=T, p=c(1,4,2))  # 1/7 like B better; 2/7 like A better
B = pmin(A+d, 5)

summary(A);  summary(B);  summary(B-A)
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  ## A
  1.000   1.000   2.000   1.943   2.000   4.000 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  ## B
    0.0     1.0     2.0     2.1     3.0     4.0 
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.  ## B-A
-1.0000  0.0000  0.0000  0.1571  1.0000  1.0000 

A Wilcoxon signed rank test on differences B - A would not work well because of the preponderance of ties and 0's. Also, there are very few different integer values in B - A so that it seems too much of a stretch to assume normality and do a t test.

A tally of B-A is as follows:

table(B-A)

-1  0  1 
 8 43 19 

That is, among those who indicated any difference between A and B, 19 out of 27 preferred A.

A 'sign test' or 'binomial test' assesses whether equally preferred products A and B would show such a distribution of lower and higher scores, the null hypothesis. In R, the procedure binom.test, with a 2-sided alternative, gives the following results:

binom.test(19, 27)

        Exact binomial test

data:  19 and 27
number of successes = 19, number of trials = 27, p-value = 0.05224
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.4981863 0.8624734
sample estimates:
probability of success 
             0.7037037 

The P-value slightly greater than 5% shows that we do not have evidence that the products differ in favorability at the 5% level. In R, the P-value is computed from the binomial CDF pbinom as shown below:

2*(1-pbinom(18, 27, .5))
[1] 0.05223899

However, suppose B is a cheapened version of A and you anticipated that B would be rated worse than A, if anything. Then a one-sided test would be appropriate, as shown below. It shows significance at the 5% level.

binom.test(19, 27, alte="g")

        Exact binomial test

data:  19 and 27
number of successes = 19, number of trials = 27, p-value = 0.02612
alternative hypothesis: true probability of success is greater than 0.5
95 percent confidence interval:
 0.5286085 1.0000000
sample estimates:
probability of success 
             0.7037037 

Notes: (a) For this illustration I have chosen fake data that shows relatively little difference between products. If there were a more substantial difference, then 70 subjects would likely show a significant difference in the two-sided test. (b) This page shows documentation for the R procedure SIGN.test (not in base R), which would accept the vector B-A as input [instead of counts derived from this vector, as for binom.test.] (c) It has been some time since I used Stata; the first thing to check is whether it implements a 'sign test' to your satisfaction. For promising links, google sign test Stata. Otherwise, you can use its binomial PDF function to find P-values.

Addendum: As I mentioned earlier, the Wilcoxon signed-rank test is of questionable value here because there are so few unique values in the difference B-A. In my example, three: $-1, 0, 1,$ mostly $0$'s, so that that nothing is lost using the sign test. In general with differences in Likert scores, there can be very few unique non-zero values unless differences of opinion about A and B are profound.

length(unique(B-A))
[1] 3

wilcox.test(B-A)

        Wilcoxon signed rank test with continuity correction

data:  B - A
V = 266, p-value = 0.03545
alternative hypothesis: true location is not equal to 0

Warning messages:
1: In wilcox.test.default(B - A) : cannot compute exact p-value with ties
2: In wilcox.test.default(B - A) :
  cannot compute exact p-value with zeroes

If there were only a few ties or 0's, maybe I'd be willing to trust the approximate value given, but this situation presents extreme cases. Then claims in a proprietary technical bulletin that all is well could hardly be persuasive.

Artificially breaking ties by slight jittering (as with wilcox.test(B-A+runif(70,-.1,.1))) gave P-values from .03 to .46 in a dozen tries, only two of them below 5%. I believe power is not being 'hemoraged'; it was never there.

Perhaps a permutation test would have better power than a sign test, but there can't be many unique medians in a permutation distribution.

$\endgroup$
7
  • $\begingroup$ The sign test hemorrhages statistical power relative to the sign rank test because it discard information about the (ranked) magnitude of the differences. $\endgroup$
    – Alexis
    Commented Aug 7, 2018 at 0:17
  • $\begingroup$ True, but here there are too many ties and 0's for the signed rank test. $\endgroup$
    – BruceET
    Commented Aug 7, 2018 at 0:19
  • 2
    $\begingroup$ Stata's signed rank test implementation adjusts for both zeros and ties out of the box. Article on it: Sribney, W. M. (1995). Correcting for ties and zeros in sign and rank tests. Stata Technical Bulletin, 26:2–4. $\endgroup$
    – Alexis
    Commented Aug 7, 2018 at 0:22
  • $\begingroup$ An exact Wilcoxon signed-rank test in the presence of ties is provided by the exactRankTests package for R. (For your example it should result in exactly the same p-value as the sign test.) Nevertheless, there are a couple of reasons one might prefer a sign test, though I think you've not quite put your finger on them. First, the null hypothesis for the Wilcoxon signed-rank test is that the ranked differences are symmetric around nought - but asymmetry needn't result from A's being better than B (or vice versa) in any sense. ... $\endgroup$
    – Scortchi
    Commented Aug 7, 2018 at 12:57
  • $\begingroup$ ... Second, to rank differences you need to be able to take differences - getting around this by first transforming "very good" &c. into ranks isn't always what you'd want to do. The sign test doesn't require a decision about whether '1' for A & '3' for B is a difference of larger or smaller magnitude than '5' for A & '4' for B. $\endgroup$
    – Scortchi
    Commented Aug 7, 2018 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.