How can I test for within-subject "consistency" across repeated measures?

Question

I have a dataset with a clear question I want to answer, but I'm unsure of the best statistical approach to answer my question. Here's an example that matches the structure of my data:

Let's say we have a group of ~200 physicians for whom we measure 10 comparable quality measures (e.g. adherence to guidelines), unit of measurement is a percent adherence to the measure. The question I want to ask is whether or not physicians tend to have consistently high or low adherence to all measures, or whether or not physicians have uneven adherence and tend to perform well on certain measures but not others. Ideally, the test or method I use can give me meaningful information for each individual.

I think that the 10 separate quality measures can be conceptualized as a "repeated measure" because they are clinically very similar and I except some within-subject correlation. I pondered repeated-measures ANOVA to test the null hypothesis that there is no significant difference across the quality measures; but I worry that it will be too sensitive and almost surely reject the null hypothesis and I won't learn very much, especially at the individual physician level.

I'd like to assign each physician to a percentile for the measures and perform some non-parametric test, but I'm getting out of my depth and was wondering if the CrossValidated community can help point me in the right direction. Suggestions for R packages also appreciated.

Thanks!

Answer 1

My guess here is that you shouldn't be looking at regression type models at all. I would start by making a scatterplot matrix of all the variables, which will give you a quick sense of whether the variables tend to hang together individually. If all of the pairwise relationships are essentially linear (this point is really important), then you can compute the correlation matrix, and submit this to a Factor Analysis. This is a nice overview of FA.

Notice that I have turned the question around, instead of asking whether physicians consistently score high or low, I'm asking whether the measures are consistent with each other. However, if the individuals rank consistently (say, third best on all measures, etc.) then the measures will end up being correlated.

It sounds to me like your main question is whether physicians in general are consistent, but it also sounds a little bit as though you would like to be able to identify the consistency / randomness of an individual physician. Update: If this latter issue is of interest, I'm less sure of what the best approach is, but here's a suggestion: You could standardize (convert to z-scores) all the measures relative to the distributions of each measure in turn (i.e., by column). Then calculate the standard deviation of that physician's z-scores (i.e., by row). Someone else may still chip in with a better idea, but this should do in a pinch. The idea is a physician who's consistently ranked (high / middling / low) should have a set of z-scores that don't vary much, whereas someone who's random will be all over the map with some high scores, some low scores etc. Presumably, the z-score SD's would be distributed as a chi-squared.