# Intra- and inter-rater reliability on the same data

I've read some of the answers here (specifically the 'Inter-rater reliability for ordinal or interval data' one). I'm still somewhat perplexed though!

I have data for 4 raters, who each rated every subject's CT scan, on two occasions, for the presence or absence of 9 signs. The two readings were separated by a washout period. As such, my data looks like this (for each sign/variable):

A1  A2  B1  B2  C1  C2  D1  D2
0   1    1   1   0   1   0   0
1   1    0   0   1   1   1   1
etc


With A1 representing the first reading by rater A, and A2 the second, and so on. Calculating the intrarater reliability is easy enough, but for inter-, I got the Fleiss Kappa and used bootstrapping to estimate the CIs, which I think is fine. Except, obviously this views each 'rating' by a given rater as being different 'raters'. I thought about using the ICC with the WinPepi function which can provide interrater agreement for fixed raters, but I suspect it might not be appropriate for binary data?

Would you just get the Fleiss and report it as such?

PS: The answer provided below by StasK is brilliant, and did clarify some things, but because of an ambiguity in the original question it isn't quite what I was looking for...

I assume that A through D are different symptoms, say, and 1 and 2 are the two raters. As you tagged this in Stata, I will build a Stata example. Let us first simulate some data: we have a bunch of subjects with two uncorrelated traits, and a battery of questions, tapping upon these traits. The two raters have different sensitivities to each of the traits: the first rater is a tad more likely than the second rater to give a positive answer on question A, but slightly less likely to give a positive answer on question B, etc.

    clear
set seed 10101
set obs 200

* generate orthogonal individual traits
generate trait1 = rnormal()
generate trait2 = rnormal()

* raters' interecepts for individual questions
local q1list 0.3 0.7 -0.2 -0.4
local q2list 0.5 0.5 0 -0.5

* prefixes
local letters a b c d

forvalues k = 1/4 {
local thisletter : word k' of letters'
local rater1     : word k' of q1list'
local rater2     : word k' of q2list'
generate byte thisletter'1 = ( k'/3*trait1 + (3-k')/5*trait2 + 0.3*rnormal() > rater1' )
generate byte thisletter'2 = ( k'/3*trait1 + (3-k')/5*trait2 + 0.3*rnormal() > rater2' )
}


This should produce something like

      . list a1-d2 in 1/5, noobs

+---------------------------------------+
| a1   a2   b1   b2   c1   c2   d1   d2 |
|---------------------------------------|
|  1    1    0    0    1    0    1    1 |
|  0    0    0    0    0    1    1    1 |
|  0    0    0    0    0    0    0    0 |
|  1    0    0    1    1    1    1    1 |
|  0    0    0    1    1    1    1    1 |
+---------------------------------------+


which I hope resembles your data, at least in terms of the existing variables.

A fully non-parametric summary of the inter-rater agreement can be constructed by converting the binary representation into a decimal representation. The outcome a1=0, b1=0, c1=0, c4=0 is 0000b=0; the outcome in the first observation is 1011b = 11, etc. Let us produce this encoding:

      generate int pattern1 = 0
generate int pattern2 = 0
forvalues k = 1/4 {
local thisletter : word k' of letters'
replace pattern1 = pattern1 + thisletter'1 * 2^(4-k')
replace pattern2 = pattern2 + thisletter'2 * 2^(4-k')
tab pattern*
}


This should produce something like

      . list  a1- d2 pat* in 1/5, noobs

+-------------------------------------------------------------+
| a1   a2   b1   b2   c1   c2   d1   d2   pattern1   pattern2 |
|-------------------------------------------------------------|
|  1    1    0    0    1    0    1    1         11          9 |
|  0    0    0    0    0    1    1    1          1          3 |
|  0    0    0    0    0    0    0    0          0          0 |
|  1    0    0    1    1    1    1    1         11          7 |
|  0    0    0    1    1    1    1    1          3          7 |
+-------------------------------------------------------------+


Now, these patterns are perfectly comparable using kap:

        . kap pattern1 pattern2

Agreement   Exp.Agrmt    Kappa     Std. Err.      Z        Prob>Z
-----------------------------------------------------------------
54.00%        17.91%     0.4396     0.0308      14.25      0.0000


You can play with the sample size or with the differences between raters to produce a non-significant answer :). This kappa suffers from a serious drawback: it does not reflect the fact of having some common items: the patterns 0001 and 0000, even though they match by 75%, would be considered non-matches within this approach. So it is an extremely conservative measure of the inter-rater agreement.

To get fair estimates of all the ICCs, you would need to run a cross-classified mixed model. Let us first reshape the data to make it possible:

        generate long id = _n

* reshape the raters
reshape long a b c d , i(id) j(rater 1 2)

* reshape the items
forvalues k = 1/4 {
local thisletter : word k' of letters'
rename thisletter' qk'
}
reshape long q , i(id rater) j(item 1 2 3 4)


Now, we can run xtmelogit (or gllamm if you like it better) on this data:

        . xtmelogit q || _all : R.rater || _all: R.item || _all: R.id, nolog

Note: factor variables specified; option laplace assumed

Mixed-effects logistic regression               Number of obs      =      1600
Group variable: _all                            Number of groups   =         1

Obs per group:
min =      1600
avg =    1600.0
max =      1600

Integration points =   1                        Wald chi2(0)       =         .
Log likelihood = -697.55526                     Prob > chi2        =         .

------------------------------------------------------------------------------
q            |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
-------------+----------------------------------------------------------------
_cons        |  -.7795316   .9384147    -0.83   0.406    -2.618791    1.059727
------------------------------------------------------------------------------

------------------------------------------------------------------------------
Random-effects Parameters    |   Estimate   Std. Err.     [95% Conf. Interval]
-----------------------------+------------------------------------------------
_all: Identity               |
sd(R.rater)                  |   .1407056   .1627763      .0145745    1.358408
-----------------------------+------------------------------------------------
_all: Identity               |
sd(R.item)                   |   1.797133   .6461083      .8882897    3.635847
-----------------------------+------------------------------------------------
_all: Identity               |
sd(R.id)                     |    3.18933   .2673165      2.706171    3.758751
------------------------------------------------------------------------------
LR test vs. logistic regression:     chi2(3) =   793.71   Prob > chi2 = 0.0000

Note: LR test is conservative and provided only for reference.
Note: log-likelihood calculations are based on the Laplacian approximation.


This is a cross-classified model with three random effects: subjects, raters and items, assuming that they are uncorrelated (which is wrong for this data; see below). Let us now estimate the ICCs:

        . local Vrater ( exp(2*_b[lns1_1_1:_cons]) )
. local Vitem ( exp(2*_b[lns1_2_1:_cons]) )
. local Vid ( exp(2*_b[lns1_3_1:_cons]) )
. nlcom Vrater' / (Vrater' + Vitem' + Vid' + _pi*_pi/3 )

-----------------------------------------------------------------------
q     |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------+----------------------------------------------------------------
_nl_1 |   .0011847   .0027384     0.43   0.665    -.0041824    .0065519
-----------------------------------------------------------------------

. nlcom Vid' / (Vrater' + Vitem' + Vid' + _pi*_pi/3 )

------------------------------------------------------------------------
q     |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------+----------------------------------------------------------------
_nl_1 |   .6086839   .0903816     6.73   0.000     .4315393    .7858285
------------------------------------------------------------------------

. nlcom Vitem' / (Vrater' + Vitem' + Vid' + _pi*_pi/3 )

-----------------------------------------------------------------------
q     |      Coef.   Std. Err.      z    P>|z|     [95% Conf. Interval]
------+----------------------------------------------------------------
_nl_1 |    .193265   .1121376     1.72   0.085    -.0265206    .4130506
-----------------------------------------------------------------------


(Hint: I figured out the names of the parameters by matrix list e(b).)

These are ICCs corresponding to raters, subjects and items, respectively. The zero ICC of the raters actually makes sense in the context of how the data were generated: there is no systematic effect in the sense that one rater consistently rates the condition better or worse than the other rater. There is an interaction between rater and item, but the model does not reflect it. True to life would be something like

 xtmelogit q ibn.item##ibn.rater, nocons || id:


With this specification, you would have to get ICCs by an even more complicated mix of the variance components and the point estimates from the fixed effects part of the model.

If you have the patience (or a powerful computer), you can specify intp(7) or something like that to get an approximation more accurate than the Laplace approximation (a single point at the mode of the distribution of the random effects).

• (+1) Impressive response. Welcome to our site! – whuber Aug 9 '11 at 1:03
• Thanks! I spent some time thinking over it... and then yet another idea came to mind, so I spent even more time thinking about it :) – StasK Aug 9 '11 at 13:42
• Wow, awesome response! Just to let you know: in Stata there's a system macro (c(alpha)) that returns all the lowercase letters (a-z). So, instead of defining the local macro letters to loop over a b c d, you can loop directly over the first 4 elements of c(alpha). (Sorry, I've noticed only now that this question is rather old). – boscovich Mar 27 '12 at 19:59