When is it more appropriate to use linear or quadratic weights to measure inter-rater reliability (Fleiss's kappa or Gwet's AC1?) [duplicate]

Question

I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not excluding a linear interpretation of results). I am trying to measure multiple inter-rater reliability using Fleiss's kappa or Gwet's AC1 but I am unsure of how appropriate it is to use weights, especially linear or quadratic weights. Both types of weights give me higher coefficients but I don't know what would justify the use of weights, let alone the use of a specific type of weight. What would be most appropriate for Likert scale ratings of a set of similar stimuli (sentences)?

Answer 1

It depends on how much partial credit you want to give for being close but wrong. Quadratic is more forgiving and gives more partial credit than linear. I give some examples here.