Measurements from two raters. Should I use multilevel/"random-effects" model? I have several variables that were measured on patients, for most patients the variables of interest were measured by 2 independent "raters". Most of the variables are binary. I need to compute the rates within each variable, i.e. the proportion of 1's. I will also need to correlate those proportion with some other "patient-level" variables.
What would make more sense: "merging" the two ratings into one (e.g. at least one "yes" is a "yes", and "no" otherwise), or to do a multilevel model a-la WinBUGS. The justification for the first option is the fact that inter-rater reliability is good.
Thanks!
 A: Merging in most cases is not a good option. Multilevel model is better idea. Of course, not a classical generalized linear mixed model, because you have not enough observations for "random" effect, but Bayesian hierarchical model would be probably fine - at least based on what I understand from your description.
You probably choose to have two independent raters for a reason, so there's no point to "drop" one rater by merging the data. It would be different if they gave 100% similar answers, than it probably wouldn't make sense, but I can imagine it is not the case. If not, then the information on when your raters vary on opinion could also be interesting and it would be good to include it in the model. Think of it - as you did - as multilevel model: group level average (or mode, or some other value) tells you about how do raters as a group rate your patients, so it does not tell you about individual raters ratings. Even if in most cases the ratings are the same: isn't interesting when they are not the same? Also choosing the criteria for "merging" raters would be subjective: why at least one "yes" and not at least one "no"?
