Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.

I've generated a user test to compare two methods: M1 and M2. I generate 40 test cases and show the result of each method on test case to 20 individuals, side by side, the individuals don't know what result came from which method. For each test case each person has to say if the result computed by M1 is better or M2 is better or they are equally good.

I want to know if M1 is better than M2. I add up all the results and generate 3-D histogram, votes for M1, votes for tie, and votes for M2.

If I only looked at M1 and M2 as 2-D histogram. I know that if M1 and M2 were equally good this histogram would be uniform. Then I'll just perform $\chi^2$ test.

What I don't know how to model are the votes for tie. Here are two options I've thought of:

  • The basis of chi-squared test is that histograms are mutually exclusive and add up to one. It seems like the votes for tie can be split in two and added to each M1 and M2 (and ties removed), but this does not seem very principled.
  • Another option is that I could just ignore the ties, that seems flawed because it breaks the "add up to one" property. For example if I had (M1:2, ties:98 M2:0) the difference between both methods would be not statistically significant.

What else can I do? Am I looking at this incorrectly? This seems like a common problem people would face when modeling user votes. What is correct way to model the ties?

share|improve this question
It sounds a lot like you are dealing with a paired preference (comparison) model, right? – chl Aug 6 '12 at 9:03
I don't understand why it's problematic that M1: 2 Ties: 98 M2: 0 should not be statistically significant. In essence, you'd have a sample of 2 people who had a preference and no such table with only 2 people would be stat. sig. – Peter Flom Aug 6 '12 at 10:14
2  
No, the two are answering different questions, so they get different answers. Dropping the ties seems to me to answer the question you want to ask – Peter Flom Aug 7 '12 at 10:56
1  
As a side note, re: "If I only looked at M1 and M2 as 2-D histogram. I know that if M1 and M2 were equally good this histogram would be uniform", this is a common misconception. The $\chi^2$ test only checks if rows & columns are independent, ie each row is similar to the other rows; they do not have to be uniform. – gung Aug 8 '12 at 13:49
2  
It's certainly true that the distribution of a fair die is a discrete uniform, & that a specific die can be tested against this "particular theoretical distribution" for fairness. But $\chi^2$ goodness of fit tests can also be conducted against other (non-uniform) theoretical distributions, & the $\chi^2$ test of independence (which you would be using) certainly does not require this. – gung Aug 8 '12 at 15:41
show 10 more comments

2 Answers

A psychologically meaningful model can guide us.

Derivation of a useful test

Any variation in the observations can be attributed to variations among the subjects. We might imagine that each subject, at some level, comes up with a numeric value for the result of method 1 and a numeric value for the result of method 2. They then compare these results. If the two are sufficiently different, the subject makes a definite choice, but otherwise the subject declares a tie. (This relates ties to the existence of a threshold of discrimination.)

The variation among the subject causes variation in the experimental observations. There will be a certain chance $\pi_1$ of favoring method 1, a certain chance $\pi_2$ of favoring method 2, and a certain chance $\pi_0$ of a tie.

It is fair to assume the subject respond independently of one another. Accordingly, the likelihood of observing $n_1$ subjects favoring method 1, $n_2$ subjects favoring method 2, and $n_0$ subjects giving ties, is multinomial. Apart from an (irrelevant) normalizing constant, the logarithm of the likelihood equals

$$n_1 \log(\pi_1) + n_2 \log(\pi_2) + n_0 \log(\pi_0).$$

Given that $\pi_0 + \pi_1 + \pi_2=0$, this is maximized when $\pi_i = n_i/n$ where $n = n_0+n_1+n_2$ is the number of subjects.

To test the null hypothesis that the two methods are considered equally good, we maximize the likelihood subject to the restriction implied by this hypothesis. Bearing in mind the psychological model and its invocation of a hypothetical threshold, we will have to live with the possibility that $\pi_0$ (the chance of ties) is nonzero. The only way to detect a tendency to favor one model over the other lies in how $\pi_1$ and $\pi_2$ are affected: if model 1 is favored, then $\pi_1$ should increase and $\pi_2$ decrease, and vice versa. Assuming the variation is symmetric, the no-preference situation occurs when $\pi_1=\pi_2$. (The size of $\pi_0$ will tell us something about the threshold--about discriminatory ability--but otherwise gives no information about preferences.)

When there is no favored model, the maximum likelihood occurs when $\pi_1=\pi_2 = \frac{n_1+n_2}{2}/n$ and, once again, $\pi_0 = n_0/n$. Plugging in the two previous solutions, we compute the change in maximum likelihoods, $G$:

$$\eqalign{ G &=\left(n_1\log\frac{n_1}{n} + n_2\log\frac{n_2}{n} + n_0\log\frac{n_0}{n}\right) \\ &-\left(n_1\log\frac{(n_1+n_2)/2}{n} + n_2\log\frac{(n_1+n_2)/2}{n} + n_0\log\frac{n_0}{n}\right) \\ &=n_1 \log\frac{2n_1}{n_1+n_2} + n_2 \log\frac{2n_2}{n_1+n_2}. }$$

The size of this value--which cannot be negative--tells us how credible the null hypothesis is: when $G$ is small, the data are "explained" almost as well with the (restrictive) null hypothesis as they are in general; when the value is large, the null hypothesis is less credible.

The (asymptotic) maximum likelihood estimation theory says that a reasonable threshold for this change is one-half the $1-\alpha$ quantile of a chi-square distribution with one degree of freedom (due to the single restriction $\pi_1=\pi_2$ imposed by the null hypothesis). As usual, $\alpha$ is the size of this test, often taken to be 5% ($0.05$) or 1% ($0.01$). The corresponding quantiles are $3.841459$ and $6.634897$.


Example

Suppose that out of $n=20$ subjects, $n_1=3$ favor method 1 and $n_2=9$ favor method 2. That implies there are $n_0 = 20 - 3 - 9 = 8$ ties. The likelihood is maximized, then, for $\pi_1 = 3/20 = 0.15$ and $\pi_2 = 9/20 = 0.45$, where it has a value of $-20.208\ldots$. Under the null hypothesis the likelihood is instead maximized for $\pi_1 = \pi_2 = 6/20 = 0.30$, where its value is only $-21.778$. The difference of $G = -20.208 - (-21.778) = 1.57$ is less than one-half the $\alpha = $5% threshold of $3.84$. We therefore do not reject the null hypothesis.


About ties and alternative tests

Looking back at the formula for $G$, notice that the number of ties ($n_0$) does not appear. In the example, if we had instead observed $n=100$ subjects and among them $3$ favored method 1, $9$ favored method 2, and the remaining $100 - 3 - 9 = 88$ were tied, the result would be the same.

Splitting the ties and assigning half to method 1 and half to method 2 is intuitively reasonable, but it results in a less powerful test. For instance, let $n_1=5$ and $n_2=15$. Consider two cases:

  1. $n=20$ subjects, so there were $n_0=0$ ties. The maximum likelihood test would reject the null for any value of $\alpha$ greater than $0.02217$. Another test frequently used in this situation (because there are no ties) is a binomial test; it would reject the null for any value of $\alpha$ greater than $0.02660$. The two tests therefore would typically give the same results, because these critical values are fairly close.

  2. $n=100$ subjects, so there were $n_0=80$ ties. The maximum likelihood test would still reject the null for any value of $\alpha$ greater than $0.02217$. The binomial test would reject the null only for any value of $\alpha$ greater than $0.3197$. The two tests give entirely different results. In particular, the $80$ ties have weakened the ability of the binomial test to distinguish a difference that the maximum likelihood theory suggests is real.

Finally, let's consider the $3 \times 1$ contingency table approach suggested in another answer. Consider $n=20$ subjects with $n_1=3$ favoring method 1, $n_2=10$ favoring method 2, and $n_0=7$ with ties. The "table" is just the vector $(n_0,n_1,n_2)=(7,3,10)$. Its chi-squared statistic is $3.7$ with two degrees of freedom. The p-value is $0.1572$, which would cause most people to conclude there is no difference between the methods. The maximum likelihood result instead gives a p-value of $0.04614$, which would reject this conclusion at the $\alpha=$5% level.

With $n=100$ subjects suppose that only $1$ favored method 1, only $2$ favored method 2, and there were $97$ ties. Intuitively there is very little evidence that one of these methods tends to be favored. But this time the chi-squared statistic of $182.42$ clearly, incontrovertibly, (but quite wrongly) shows there is a difference (the p value is less than $10^{-15}$).

In both situations the chi-squared approach gets the answer entirely wrong: in the first case it lacks power to detect a substantial difference while in the second case (with lots of ties) it is extremely overconfident about an inconsequential difference. The problem is not that the chi-squared test is bad; the problem is that it tests a different hypothesis: namely, whether $\pi_1=\pi_2=\pi_0$. According to our conceptual model, this hypothesis is psychological nonsense, because it confuses information about preferences (namely, $\pi_1$ and $\pi_2$) with information about thresholds of discrimination (namely, $\pi_0$). This is a nice demonstration of the need to use a research context and subject matter knowledge (however simplified) in selecting a statistical test.

share|improve this answer
You say "Looking back at the formula for G, notice that the number of ties (n0) does not appear"... but I see n0 as a term in the formula for change in log-lik. Is that not G? – Russell S. Pierce Aug 10 '12 at 14:32
1  
@dr See the final expression for $G$: it is the difference in log likelihoods. Although both likelihoods depend on $n_0$, cancellation removes that dependence altogether. – whuber Aug 10 '12 at 14:42
Ah, I see now. I missed the equals sign that indicated the reduction of the equation. – Russell S. Pierce Aug 10 '12 at 14:43
How does your solution compare to a 2x2 contingency table approach excluding ties? – Russell S. Pierce Aug 10 '12 at 14:45
1  
@dr It should be identical. The point of this derivation was to justify this approach using basic principles of statistical inference and psychology, because it appears that the crux of the issue concerns the right way to handle the ties. – whuber Aug 10 '12 at 14:51

I suspect whuber's answer is (as usual) more replete than what I am about to type. I admit, I may not fully understand whuber's answer... so what I am saying may not be unique or useful. However, I did not notice where in whuber's answer the nesting of preferences under individuals as well as the nesting of preferences within test-cases was considered. I think given the question asker's clarification that:

The cases are indeed a random sample of all possible cases. I think an analogy is the following: the election is determined by what happens at the polls, but I do have for each voter their party affiliation. So it would be almost expected that a candidate from one party appeals to the voters affiliated with that party, but this is not necessarily a given, a great candidate can win in his party and win over people form the other party.

... these are important considerations. Therefore, perhaps what is most appropriate is not $\chi^2$ but a multi-level logistic model. Specifically in R I might cast something like:

lmer(PreferenceForM1~1+(1|RaterID)+(1|TestCaseID),family=binomial)

PreferenceForM1 would be coded as 1 (yes) and 0 (no). Here an intercept over 0 would indicate an average rater's preference for method 1 on an average test case. With samples near the lower bounds of usefulness for these techniques, I'd probably also use pvals.fnc and influence.ME to investigate my assumptions and the effects of outliers.

The basic question about ties here seems well answered by whuber. However, I'll (re-)state that it seems that ties reduce your ability to observe a statistically significant difference between the methods. In addition, I'll claim that eliminating them may cause you to over-estimate the preference individuals have for one method versus the other. For the later reason, I'd leave them in.

share|improve this answer
I'm probably confused by the R notation, but doesn't your suggestion have more parameters than there are data? This confusion is not of your making: I had understood that there were $20$ subjects and just one result per subject (M1, M2, or tie), but the interpretation implicit in your answer is that there may be $800=20*40$ observations. Carlosdc, could you clarify this for us? – whuber Aug 10 '12 at 14:49
OP stated that "I generate 40 test cases and show the result of each method on test case to 20 individuals";"For each test case each person has to say if the result computed by M1 is better or M2 is better or they are equally good." So, I was interpreting OP as saying there were 20 * 40 observations. – Russell S. Pierce Aug 10 '12 at 14:53
You're right, there would be a lot of parameters being estimated in this data. The exact number I'm fuzzy on (a place where the stats package has allowed me to get complacent with my understanding of the underlying equations). – Russell S. Pierce Aug 10 '12 at 14:55

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.