Reliability in Elicitation Exercise

I am trying to calculate the reliability in an elicitation exercise by analysing some test-retest questions given to the experts. The experts elicited a series of probability distributions which were then compared with the true value (found at a later date) by computing the standardized quadratic scores. These scores are the values that I am using to calculate the reliability between the test-retest results.

Which reliability method would be appropriate here? I was looking mostly at Pearson's correlation and Chronbach's alpha (and got some negative values using both methods) but I am not sure this is the right approach.

UPDATE: Background information

The data were collected from a number of students who were asked to predict their own actual exam mark in four chosen modules by giving a probability distribution of the marks. One module was then repeated at a later date (hence the test-retest exercise).

Once the exam was taken, and the real results were available, the standardized quadratic scores were computed. These scores are proper scoring rules used to compare assessed probability distributions with the observed data which might be known at a later stage.

The probability score Q is defined as:

where k is the total number of elicited probabilities and j is the true outcome.

My question is which reliability method would be more appropriate when it comes to assessing the reliability between the scores of the repeated modules? I calculated Pearson's correlation and Chronbach's alpha (and got some negative values using both methods) but there might be a better approach.

• Is your your probability score identical to what is called the Brier score? (See: en.wikipedia.org/wiki/Brier_score)
– user28
Jul 31, 2010 at 4:34
• @Srikant: Yes it is. It is known by both names. Jul 31, 2010 at 6:49

Maybe I misunderstood the question, but what you are describing sounds like a test-retest reliability study on your Q scores. You have a series of experts each going to assess a number of items or questions, at two occasions (presumably fixed in time). So, basically you can assess the temporal stability of the judgments by computing an intraclass correlation coefficient (ICC), which will give you an idea of the variance attributable to subjects in the variability of observed scores (or, in other words of the closeness of the observations on the same subject relative to the closeness of observations on different subjects).

The ICC may easily be obtained from a mixed-effect model describing the measurement $y_{ij}$ of subject $i$ on occasion $j$ as

$$y_{ij}=\mu+u_i+\varepsilon_{ij},\quad \varepsilon\sim\mathcal{N}(0,\sigma^2)$$

where $u_i$ is the difference between the overall mean and subject $i$'s mean measurement, and $\varepsilon_{ij}$ is the measurement error for subject $i$ on occasion $j$. Here, this is a random-effect model. Unlike a standard ANOVA with subjects as factor, we consider the $u_i$ as random (i.i.d.) effects, $u_i\sim\mathcal{N}(0,\tau^2)$, independent of the error terms. Each measurement differ from the overall mean $\mu$ by the sum of the two error terms, among which the $u_i$ is shared between occasion on the same subjects. The total variance is then $\tau^2+\sigma^2$ and the proportion of the total variance that is accounted for by the subjects is

$$\rho=\frac{\tau^2}{\tau^2+\sigma^2}$$

which is the ICC, or the reliability index from a psychometrical point of view. Note that this reliability is sample-dependent (as it depends on the between-subject variance). Instead of the mixed-effects model, we could derive the same results from a two-way ANOVA (subjects + time, as factors) and the corresponding Mean Squares. You will find additional references in those related questions: Repeatability and measurement error from and between observers, and Inter-rater reliability for ordinal or interval data.

In R, you can use the icc() function from the psy package; the random intercept model described above corresponds to the "agreement" ICC, while incorporating the time effect as a fixed factor would yield the "consistency" ICC. You can also use the lmer() function from the lme4 package, or the lme() function from the nlme package. The latter has the advantage that you can easily obtain 95% CIs for the variance components (using the intervals() function). Dave Garson provided a nice overview (with SPSS illustrations) in Reliability Analysis, and Estimating Multilevel Models using SPSS, Stata, SAS, and R constitutes a useful tutorial, with applications in educational assessment. But the definitive reference is Shrout and Fleiss (1979), Intraclass Correlations: Uses in Assessing Rater Reliability, Psychological Bulletin, 86(2), 420-428.

I have also added an example R script on Githhub, that includes the ANOVA and mixed-effect approaches.

Also, should you add a constant value to all of the values taken at the second occasion, the Pearson correlation would remain identical (because it is based on deviations of the 1st and 2nd measurements from their respective means), whereas the reliability as computed through the random intercept model (or the agreement ICC) would decrease.

BTW, Cronbach's alpha is not very helpful in this case because it is merely a measure of the internal consistency (yet, another form of "reliability") of an unidimensional scale; it would have no meaning should it be computed on items underlying different constructs. Even if your questions survey a single domain, it's hard to imagine mixing the two series of measurements, and Cronbach's alpha should be computed on each set separately. Its associated 95% confidence interval (computed by bootstrap) should give an indication about the stability of the internal structure between the two test occasions.

As an example of applied work with ICC, I would suggest

Johnson, SR, Tomlinson, GA, Hawker, GA, Granton, JT, Grosbein, HA, and Feldman, BM (2010). A valid and reliable belief elicitation method for Bayesian priors. Journal of Clinical Epidemiology, 63(4), 370-383.

You would use Cronbach alpha if you do not know the true value but if you do know the true value then it seems a bit pointless to use Cronbach alpha. The use of Pearson correlation also seems a bit odd as you do not actually have a paired set of values. I would suggest using something like the Mean Squared Error (MSE). Suppose that you have N experts and that the expected estimate for the expert i is given by $\hat{\theta_i}$ and your true value is $\theta$. Then,

$MSE = \frac{\sum_i (\hat{\theta_i} - \theta)^2}{N}$

• Thanks for your thoughts. Maybe I wasn't clear though. I do have paired sets. They are the standardized quadratic scores of the test-retest results, ie some distributions were elicited twice. I will edit the question. Jul 30, 2010 at 6:42
• Could you clarify how you are computing standardized quadratic scores? And, what exactly your are correlating?
– user28
Jul 30, 2010 at 14:44
• @Srikant: I updated the question with some background information Jul 30, 2010 at 17:16