2
$\begingroup$

On crowd sourcing sites like mechanical turk, we may ask people to rate two models in the scale of, say 1 to 5.

Obviously, some people will give 5 for the good one and 1 for the bad one, while some may give 3 and 2, etc. In other words, each person has their own scale within the specified range.

As such, standard deviation turns out to be large.

Let's say we ended up with ratings for two models, collected from 5000 people where each people rated both models. We have

average 3.35 with std 1.0 for model A, and average 3.17 with std 1.08 for model B.

Since the ratings for A and B significantly overlap when applying avg+/-std for the models, I am not sure whether A's superiority is statistically significant.

So my question is, could such results as described above be considered significant despite large std? If so, should other metrics like p-value be introduced to make the significance more clear?

===========edit======

I am now grouping the results into three cases, where A was rated better than B, equal to B, or worse than B.

How would the situation change?

$\endgroup$
6
  • 3
    $\begingroup$ Have you considered other types of analysis that might be better suited to a 1-5 scale? Also, note that statistical comparisons of mean values are based on standard errors of the mean, not standard deviations. With 500 evaluations of each of the 2 models, the standard errors of the means would be about 1/22 of the standard deviations. $\endgroup$
    – EdM
    Commented Jan 30, 2017 at 17:58
  • $\begingroup$ @EdM All 1000 people are rating both models, not splitting. Would calculation of standard errors of the mean still be the way to go as such? Also, how did you get to 1/22 conclusion? $\endgroup$ Commented Jan 30, 2017 at 18:05
  • $\begingroup$ Standard error of the mean is the standard deviation divided by the square root of the number of observations. Square root of 500 is approximately 22. The answer from @MarcusMorrisey covers your question well. With paired observations you should typically do some type of paired test, as that answer notes. $\endgroup$
    – EdM
    Commented Jan 30, 2017 at 18:51
  • $\begingroup$ My thought is that I would include questions that benchmark whether the rater is a 1-5 or a 2-4 person. $\endgroup$ Commented Jan 30, 2017 at 18:53
  • $\begingroup$ @EdM I would get about 3.35 mean and 0.01 std error of mean for Model A and 3.17 mean and 0.02 std error of mean for Model B. How would I interpret it? $\endgroup$ Commented Jan 30, 2017 at 19:16

1 Answer 1

4
$\begingroup$

Presuming you are referring to standard null hypothesis statistical testing then I would say you must calculate a p-value as this is what you compare to your alpha criterion to determine if a statistically significant difference exists. The simplest parametric test of the null hypothesis that Model A = Model B would be a t-test. T-tests are relatively robust to differences in standard errors. Welch's t-test, or the unequal variances t-test, proceeds from the assumption that your standard errors do in fact differ.

However, as @EdM mentions in the comments, you may want to consider different analyses that are better suited to the data. Typically a rating score like the one you describe is on an ordinal scale. Because ordinal data has may not have equal intervals between the ratings, it is not reasonable to perform multiplication and division on those values, as is required to calculate a mean or standard error. Your scale would need to be interval or ratio to carry out those calculations (see: http://psychology.okstate.edu/faculty/jgrice/psyc3214/Stevens_FourScales_1946.pdf). Instead, you might want to consider a test that can be meaningfully applied to ranks such at the Mann-Whitney U-test.

Having said that, there are those that advocate for a t-test in these cases despite the violation of its assumptions, noting that performance is often very similar, e.g.

De Winter, J. C., & Dodou, D. (2010). Five-point Likert items: t test versus Mann-Whitney-Wilcoxon. Practical Assessment, Research & Evaluation, 15(11), 1-12.

Zimmerman DW, Zumbo BN. 1993 . Rank transformations and the power of the Student t -test and Welch t ′-test for non-normal populations. Can J Exp Psychol 47 : 523 –39.

Edit: I note from your comment that your data is paired (i.e., you measure each individual twice). You would want to ensure to use a test that accounts for that relationship, a paired samples t-test if you go the parametric route or the Wilcoxon Signed Rank test for non-parametric.

$\endgroup$
10
  • $\begingroup$ Thanx! I have another question. I organized the ratings into 3 categories, A>B, A=B, A<B, which are 2208, 1222, 1570 respectively in my real case. Would your answer still hold? $\endgroup$ Commented Jan 30, 2017 at 18:51
  • $\begingroup$ I'm not sure I follow exactly. Let's step back a minute. Can you tell me what you want to know from this data in plain words. For example, are you trying to show they are different, or that one is larger? Ideally, you would edit that into your question for additional clarity. $\endgroup$ Commented Jan 30, 2017 at 18:56
  • $\begingroup$ Sorry for unclarity. By A>B, I mean to say the number of ratings where A received better score than B, whether it'd be 5 to 1 or 4 to 3. Such counts were 2208 out of 5000. I would edit it immediately. $\endgroup$ Commented Jan 30, 2017 at 18:59
  • 1
    $\begingroup$ Helpful, but the treatment of measurement scales is problematic. I can't muiltiply or divide Celsius and Fahrenheit temperatures either but i have no difficulty calculating means on those scales. It's common advice that taking a mean requires at most interval scale, not ratio scale. That said, many texts and courses tell you further that you can't or shouldn't take means of ordinal scales and then proceed to advise procedures that do exactly, e.g. Spearman correlation is based on means and variances of ranks. $\endgroup$
    – Nick Cox
    Commented Jan 30, 2017 at 19:02
  • $\begingroup$ With the data I provided for 3 categories, I calculated chi-squared test, and got 133.37, which corresponds to very low p-value. Does it mean the result is significant, or am I doing something completely wrong? $\endgroup$ Commented Jan 30, 2017 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.