2
$\begingroup$

I'd like to test people's preference on model A against three other models B,C,D.

I asked 5000 people on crowd sourcing to rate all 4 models (thus, there are 20,000 ratings overall) in the scale of 1 to 5.

I've grouped the results into 3 categories, which are 1) model A won, 2) model A lost, and 3) model A and compared model received equal rating. Results are as following:

compared model  | A won |  equal  | A lost
B                  2208    1222     1570
C                  2970    538      1492
D                  1890    1454      1656

I calculated chi-square with 2 degrees of DOF, with expected value as 2,000 for winning and losing, and 1,000 for par, and got chi-squares of 133.37, 812.93, 271.33 for each comparison.

Is my calculation correct? And if so, most online calculators simply return "p-value is less than 0.000001", but is there a way to get specific p-values from chi-squares, however small they may be? Also, since chi-squares are very high and p-values very small (if correct), does it mean that the results are statistically significant?

======edit=======

Null hypothesis is that "all models are equally preferred".

Alternative hypothesis obviously is that model A is better.

========edit=====

I grouped the results into 3 categories, because each worker has a different range of ratings. For example, worker A may give 5 to good models, and 1 to bad models, while worker B may give 3 to good models and 2 to bad models, etc. Thus, std. dev turned out to be pretty large (over +/- 1.0) and I needed to simplify the results to figure out the significance.

$\endgroup$
10
  • $\begingroup$ It's not clear what the null and alternative hypotheses are here $\endgroup$
    – Glen_b
    Jan 31, 2017 at 1:58
  • $\begingroup$ If the question is whether or not the distribution is the same across rows, no, it's not. At all. I get a $\chi^2$ around 700 on 4 degrees of freedom. $\endgroup$
    – gammer
    Jan 31, 2017 at 3:31
  • 1
    $\begingroup$ Your null hypothesis is not sufficiently specific to conduct a test. If it were modified to "all models are equally preferred," that would imply equal rates of preferences. The basis of choosing 2000 and 1000 in your table is obscure: it does not appear to be deducible from your incomplete hypothesis. $\endgroup$
    – whuber
    Feb 3, 2017 at 0:17
  • 1
    $\begingroup$ Why did you group the data from your 5 point scale into 3 categories? That is likely to reduce the power of your test. $\endgroup$
    – Joel W.
    Feb 3, 2017 at 0:21
  • 3
    $\begingroup$ it's premature to place a bounty on a question that has many issues that remain unclear. It might draw an answer but answers to unclear questions may be unhelpful or even lead you away from a suitable understanding. $\endgroup$
    – Glen_b
    Feb 3, 2017 at 1:00

2 Answers 2

3
+50
$\begingroup$

No, your use of a chisquare test is inappropriate. One problem is that the same 5000 people appear more than once in your table, so the counts are not independent. Another problem is the use of arbitrarily chosen expected values.

One simple and correct way to test your hypotheses is to compare A with each of the other models one at a time. Comparing A to B, your table shows that 2208 people prefer A and 1570 prefer B. An exact one-sided p-value can be obtained from the binomial probability in R:

pbinom(2208-0.5, prob=0.5, size=2208+1570, lower.tail=FALSE)

which is 1.4e-25. This tests the null hypothesis that people are equally likely to prefer A or B vs the alternative that people are more likely to prefer A. The binomial test conditions on the total number of people who had a preference.

Comparing A to C gives:

pbinom(2970-0.5, prob=0.5, size=2970+1492, lower.tail=FALSE)

which is 1.1e-110. Comparing A to D gives:

pbinom(1890-0.5, prob=0.5, size=1890+1656, lower.tail=FALSE)

which is 4.5e-05.

So, yes, there is strong evidence that the respondents tend to prefer model A over each of the other three alternatives.

$\endgroup$
0
2
$\begingroup$

is there a way to get specific p-values from chi-squares?

Yes. In R use 1-pchisq(), for example 1-pchisq(133.37,2) is 0. This is because the number is so small, R rounds it to 0.

Also, since chi-squares are very high and p-values very small (if correct), does it mean that the results are statistically significant?

Yes. A rule of thumb provided by Ronald Fisher is that if the p-value is less than 5% (0.05) then the result is significant.

Is my calculation correct?

I wouldn't have calculated it like this:

  1. Start by plotting the percentage of ratings for each model side-by-side
  2. Compare the shapes using a chi-square test, i.e. A vs. B, A vs. B, A vs. C. To do this, treat the A count as being the observed data, and the B data as being expected. Note the degrees of freedom would change to 5, therefore you use a chi-square(4) distribution

Some pseudo-code for point 2 in R is:

A = 5000*c(0.2,0.3,0.5,0.3,0.2) # These are the proportions rating A as 1-5
B = 5000*c(0.2,0.2,0.2,0.2,0.2) # These are the proportions rating B as 1-5
TestStat = sum(((A-B)^2)/B) # Find the chi-square test statistic (2750)
PVal = 1-pchisq(TestStat,4) # Find the p-value (0), which is significant
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.