I'm performing a study on inter-doctor variation. For that, I'm studying a diagnostic instrument that has around 100 items that all have to be rated. Some are dichotomous, some ordinal. Several thousands of patients have been scored, each of them by one rater only. I have about 200 raters.
My hypothesis is that due to the content of the items, some items will show a higher dispersion than others because the interpretation and scoring of these items is 'harder', i.e. there's a higher level of uncertainty among the doctors on how to score these items.
I calculated coefficients of variation (CV) for each item to be able to compare dichotomous and ordinal items.
Now I'm not sure how to formally test for differences in CV across these items. I know of tests for equality of variance, like bartlett's, levene, brown-forsythe. However, using these, the result would only tell me that not all variances can be considered to be equal. It doesn't tell me which items have a deviating variance (and then: deviating from what exactly?).
Three questions that I have:
- Is there a way to formally test which items have a deviating (bigger) CV?
- Related to 1, is there any way I could establish a valid cutt-of point or 'base rate CV'?
- Related to 2, is it statistically meaningful and methodologically valid to compare the CV of each item against the mean CV of all other items?