The limitation of Cohen's Kappa is that it is meant to compare two raters with usually two categories. A weighted version of Cohen's Kappa is available that allows you to expand the two category formulation, but the issue of multiple raters is largely unexplored in the statistical world.
The bigger issue is that Cohen's Kappa applies to ratings in which there is some latent notion of a "true rating" which is unobserved by you, the statistician. An example of ratings is the presence/absence of cancer in a mammographic screen by two radiologists, or depressive symptoms in an elderly patient evaluated by two psychologists... basically ratings have their stakes in diagnostic medicine. An entirely different tool is used for "gold standard evaluation" when the actual value of the screen or test is known.
If these Likert scales represent, say, opinions or sentiments as on a questionnaire, you would not consider these "ratings" though that term is consistent with the conventional vernacular. They are simply responses.
Coincidentally, there are other measures of consistency (or internal validity) in survey design that you might consider. The Cronbach's Alpha is a measure of how two people, all things similar, might be expected to report in a similar way on a survey. But this application has the alternate application of survey validation.
Bearing that in mind, perhaps there is something more you can say about what the purpose of the survey is.
