I am analysing the results of a linguistic acceptability judgment study where 36 speakers had to rate a finite number of sentences on a 7-point Likert scale (so an ordinal scale, although I am not excluding a linear interpretation of results). I am trying to measure multiple inter-rater reliability using Fleiss's kappa or Gwet's AC1 but I am unsure of how appropriate it is to use weights, especially linear or quadratic weights. Both types of weights give me higher coefficients but I don't know what would justify the use of weights, let alone the use of a specific type of weight. What would be most appropriate for Likert scale ratings of a set of similar stimuli (sentences)?
It depends on how much partial credit you want to give for being close but wrong. Quadratic is more forgiving and gives more partial credit than linear. I give some examples here.