Basic Inter-rater reliability and type of data

Question

I am looking to find out the type of data I have and the ideal test to use calculate inter-rater reliability.

Research question: Functional Movement Screening: How reliable are we in our investigations?
Aim: to look into the inter-rater reliability of Physiotherapists performing a functional movement screen (12 tests) on 2 athletes.

My current data collection includes:
6 raters each scoring 12 tests with a yes and no answer.
There are 2 sets (2 athletes) in total and each rater has given me y and n data for both.

Example:

Rater 1   6 Yes / 6 No  
Rater 2   7 Yes / 5 No  

Test 1    3 yes / 3 no (from raters)
Test 2    4 yes / 2 no

I would like to know:
1. the type of data I have?
2. am I looking at the reliability of the tests or do I need to focus on the raters scores?
3. the groups I need to organise the data into to give me the results?
4. the statistical test that would give me the inter-rater reliability of the raters and the tests?

I am afraid statistics isn't one of my strong points and so apologies if this is very basic but any help anyone could give me would be much appreciated as I am having difficult at present.

Answer 1

I'm not sure I understand questions 1 and 4, but I think I can help you out with 2 and 3!

Regarding your second question, the type of analysis you're talking about is an assessment of the inter-rater reliability. You might also be interested in the reliability of your test as well, but, given the data you presented, I don't think you can really address that question here. If I were exploring test reliability, I would do one of two things:

• Compare raters scores on the test of interest versus some gold-standard test that is well-established in the literature. This is the approach researchers will often take when they are interested in establishing the reliability of a new type of assessment.
• Compare raters scores on the test of interest over time. One would take this approach in the case that the question of interest pertains to the temporal reliability of the measure in question (i.e., whether or not raters' answers change over time or repeated assessment). This is the sort of experiment that is often performed in the early stages of establishing the reliability of neuropsychological assessments over repeated administrations, such as the Trail Making Test.

With regard to your third question, I would use Cohen's kappa. The linked wikipedia article is a pretty good reference on this, but, briefly, the kappa is a well-established measure of inter-rater agreement that quantifies the divergence from expected agreement that you observe in your data.