how do i test inter observer reliability in rating steps of a surgical procedure scored from 1-5 by 2 observers I have completed a project where in a colleague and i have rated steps in a surgery from 1-5. now we need to know the inter observer reliability. I used Cohen's kappa and got a poor score although the excel shows similar points. Please guide me as to  how i can test this in a better way
 A: This doesn't give you a technical answer but I hope these comments are useful. 
It is well known by almost everyone that Cohen's kappa is not very useful for hypothesis testing though sometimes it is useful for description. 
If you have a lot of such kappas from different scales you can look at them and notice that some of them are peculiarly large or, more likely, small.
If your ratings are ordinal, then Kendall's tau might be a better description of whether raters agree. (Kappa assumes nominal scaling, that is, there is no ordering.) Again, if you have a large number of ratings you can qualitatively evaluate whether some are much worse.
I have found the combination of Kendall's tau and the agreement rate to be informative in some cases. 
The larger problem is whether two raters can tell you much about reliability at all. The only thing you can learn is whether the two agree or disagree and how often. This is neither very interesting or informative.
If you want to know anything about the rating process, that is, the classification in other circumstances then you need more raters and a variety of items to rate.
Three raters are still inadequate, however, since this only tells you if the three mostly agree or if one of them disagrees with the other two and how often. A reasonable test of reliability would require at least five raters and preferably more. One hypothesis of interest is whether one or a few raters are distinctly different from the rest.
There is a literature on some alternative methods of testing when you have multiple raters. 
