6
$\begingroup$

I have read that Kendall W should be avoided when it comes to deal with non-rankings especially for rating scales which tend to have a lot of ties. Yet posts here seem to suggest it for ratings. As stated in this post I have a small study of 21 respondents, who rated some items from 0-5, with 0 being unimportant and 5 being very important and I'm looking for measures of agreement for specific respondents. I am not looking for Absolute agreement.

Whilst ICC was suggested as a possible solution, there is an issue with the use of the F test in this case, given the small number of respondents.

What are your views on Kendall W in this case?

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Where have you read that? A succinct description of the argument or at least a specific reference would probably be useful. $\endgroup$
    – Gala
    Commented Jul 22, 2013 at 9:36
  • $\begingroup$ Its an issue to do with a lot of ties. See the discussion here @GaëlLaurans $\endgroup$
    – Cesare Camestre
    Commented Jul 22, 2013 at 9:43
  • $\begingroup$ Added links to relevant posts @GaëlLaurans, I hope this helps in understanding further the problem. I was tempted to use the Kendall W because it was previously used in a similar case, but there are too many ties. $\endgroup$
    – Cesare Camestre
    Commented Jul 22, 2013 at 9:56
  • $\begingroup$ I agree with Gaël that the null is absurd, but if you nevertheless need a p-value then you can get one for the ICC (which is definitely preferable to W) by doing a permutation test: randomly permute each subject's ratings, then recompute the ICC; do this a few thousand times, and see where your actual ICC comes in the distribution of random ICCs. $\endgroup$
    – Ray Koopman
    Commented Oct 20, 2013 at 22:20

1 Answer 1

Reset to default
1
$\begingroup$

I don't have anything specific to say about Kendall's W but I don't get this concern about the ICC, the F test and the sample size.

Your sample is not so small that testing would necessarily be impossible but why would you want to do such a test? To see if agreement is different from 0? This is quite a low bar and should be evident from the data. If you have doubts about that, these ratings certainly don't form a good measure of anything the raters agree on so worrying about which specific measure of inter-rater of agreement you are using and the niceties of the relevant tests would not really be your main concern.

On the other hand, anything you compute on a sample this small will obviously be subject to a lot of sampling variability and uncertainty. It's a rather basic fact that has nothing to do with ICC or the F-test specifically and there is no miracle inter-rater agreement index that would allow you to go around that.

At the end of the day, I think the underlying issue is that you seem to be asking many rather abstract questions in search for the “true” inter-rater agreement and some sort of fail/pass test that would tell you if it is “good enough”. Such a thing simply does not exist in my opinion and published threshold are really quite arbitrary. Instead of trying to interpret every bit of advice recommending one index or another, I think it could be more fruitful to read broadly about inter-rater agreement measures (see the references provided in other questions on this topic) and think about what each of them reveal about your data rather than focus solely on whether agreement is “good” or not.

Improve this answer
$\endgroup$
15
  • $\begingroup$ What I am after - "Did most respondents answer the questions in the same way?". Did most respondents rate the answers in a similar way - i.e. they all rated them high / low depending on the question. That is the question $\endgroup$
    – Cesare Camestre
    Commented Jul 22, 2013 at 11:11
  • $\begingroup$ My point is that this is not a well-defined question that could be addressed with a single technique, at least outside of some obvious cases (e.g. everybody giving the exact same rating). It admits several answers, depending on whether you are interested in absolute ratings, if you want to know if participants tend to rate particular objects higher or lower, if you are afraid of certain patterns of random responses, etc. The answer will also be quantitative and not binary. $\endgroup$
    – Gala
    Commented Jul 22, 2013 at 11:44
  • 1
    $\begingroup$ Sample size doesn't affect the true distribution (but it might limit your ability to detect the true distribution). What it will do is reduce statistical power, but the F-test is showing significant agreement, and every other test you try will probably also show significant agreement, because that's what the data show. The problem with the F-test is not the sample size, it's the non-continuous and closed-ended (ordinal) rating system (or that's how it seems to me). But every test has potential problems - you have to decide. Personally, I'd just present both sets of results. $\endgroup$
    – atrichornis
    Commented Jul 23, 2013 at 13:57
  • 1
    $\begingroup$ @CesareCamestre Forget the F-test, then. $\endgroup$
    – Gala
    Commented Jul 23, 2013 at 14:23
  • 1
    $\begingroup$ Like I said it's an extremely low bar, something that is usually obviously true because the null hypothesis is absurd. I am still not sure I get precisely what it is your are doing but is it plausible that your participants would have absolutely no agreement on what's important or not? If not, why test it? (Routinely putting a p-value next to any number, no matter what, is not relevant.) $\endgroup$
    – Gala
    Commented Jul 23, 2013 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.