I've got a dataset with the following format.

Forecasts of cancer no cancer

There's a binary outcome cancer/no cancer. Every doctor in the dataset has seen every patient and given an independent judgment on whether the patient has cancer or not. The doctors then give their confidence level out of 5 that their diagnosis is correct, and the confidence level is displayed in the brackets.

I've tried various ways to get good forecasts out of this dataset.

It works pretty well for me to just average across the doctors, ignoring their confidence levels. In the table above that would have produce correct diagnoses for Patient 1 and Patient 2, although it would have incorrectly said that Patient 3 has cancer, since by a 2-1 majority the doctors think Patient 3 has cancer.

I also tried a method in which we randomly sample two doctors, and if they disagree with each other then the deciding vote goes to whichever doctor is more confident. That method is economical in that we don't need to consult a lot of doctors, but it also increases the error rate quite a bit.

I tried a related method in which we randomly select two doctors, and if they disagree with each other we randomly select two more. If one diagnosis is ahead by at least two 'votes' then we resolve things in favor of that diagnosis. If not, we keep sampling more doctors. This method is pretty economical and doesn't make too many mistakes.

I can't help feeling that I'm missing some more sophisticated way of doing things. For instance, I wonder if there is some way I could divide the dataset into training and test sets, and work out some optimal way to combine the diagnoses, and then see how those weights perform on the test set. One possibility is some sort of method that lets me downweight doctors who kept making mistakes on the trial set, and perhaps upweight diagnoses that are made with high confidence (confidence does correlate with accuracy in this dataset).

I've got various datasets matching this general description, so the sample sizes vary and not all the datasets relate to doctors/patients. However, in this particular dataset there are 40 doctors, who each saw 108 patients.

EDIT: Here is a link to some of the weightings that result from my reading of @jeremy-miles's answer.

  1. Unweighted results are in the first column. Actually in this dataset the maximum confidence value was 4, not 5 as I mistakenly said earlier. Thus following @jeremy-miles's approach the highest unweighted score any patient could get would be 7. That would mean that literally every doctor asserted with a confidence level of 4 that that patient had cancer. The lowest unweighted score any patient could get is 0, which would mean that every doctor asserted with a confidence level of 4 that that patient did not have cancer.

  2. Weighting by Cronbach's Alpha. I found in SPSS that there was an overall Cronbach's Alpha of 0.9807. I tried to verify that this value was correct by calculating Cronbach's Alpha in a more manual way. I created a covariance matrix of all 40 doctors, which I paste here. Then based on my understanding of the Cronbach's Alpha formula $\alpha = \frac{K}{K-1}\left(1-\frac{\sum \sigma^2_{x_i}}{\sigma^2_T}\right)$ where $K$ is the number of items (here the doctors are the 'items') I calculated $\sum \sigma^2_{x_i}$ by summing all the diagonal elements in the covariance matrix, and $\sigma^2_T$ by summing all the elements in the covariance matrix. I then got $\alpha = \frac{40}{40-1}\left(1-\frac{8.7915}{200.7112}\right)=0.9807$ I then calculated the 40 different Cronbach Alpha results that would occur when each doctor got removed from the dataset. I weighted any doctor who contributed negatively to Cronbach's Alpha at zero. I came up with weights for the remaining doctors proportional to their positive contribution to Cronbach's Alpha.

  3. Weighting by Total Item Correlations. I calculate all the Total Item Correlations, and then weight each doctor proportional to the size of their correlation.

  4. Weighting by Regression Coefficients.

One thing I'm still not sure about is how to say which method is working "better" than the other. Previously I had been calculating things like the Peirce Skill Score, which is appropriate for instances in which there is a binary prediction and a binary outcome. However, now I have forecasts ranging from 0 to 7 instead of 0 to 1. Should I convert all the weighted scores > 3.50 to 1, and all the weighted scores < 3.50 to 0?

  • $\begingroup$ Can we say that No Cancer (3) is Cancer (2)? That would simplify your problem a bit. $\endgroup$
    – Wayne
    Dec 23, 2016 at 15:52
  • 1
    $\begingroup$ Re: your data structure, it's almost always better to have different variables (whether patient has cancer; how confident the assessment is) in different columns. Combining them as in "no cancer (3)" severely limits your options. $\endgroup$
    – rolando2
    Dec 24, 2016 at 0:48
  • $\begingroup$ @Wayne The data ranges from the prediction of cancer with maximum confidence Cancer (4) to the prediction of no cancer with maximum confidence No Cancer (4). We can't say that No Cancer (3) and Cancer (2) are the same, but we could say there is a continuum, and the middle points in this continuum are Cancer (1) and No Cancer (1). $\endgroup$ Dec 24, 2016 at 7:08
  • $\begingroup$ @rolando2 Thanks for the advice. I've rearranged things in my own data file so that now they are separated out. $\endgroup$ Dec 24, 2016 at 7:10
  • 1
    $\begingroup$ Note that your threshold is a tunable parameter, so the appropriate cutoff will depend on your evaluation criterion. As I was unfamiliar with your metric I Googled it, and actually the first hit may be relevant to you: A note on the maximum Peirce skill score (2007). $\endgroup$
    – GeoMatt22
    Dec 24, 2016 at 8:58

4 Answers 4


First, I would see if the doctors agree with each other. You can't analyze 50 doctors separately, because you'll overfit the model - one doctor will look great, by chance.

You might try to combine confidence and diagnosis into a 10 point scale. If a doctors says that the patient doesn't have cancer, and they are very confident, that's a 0. If the doc says they do have cancer and they are very confident, that's a 9. If they doc says they don't, and are not confident, that's a 5, etc.

When you're trying to predict, you do some sort of regression analysis, but thinking about the causal ordering of these variables, it's the other way around. Whether the patient has cancer is the cause of the diagnosis, the outcome is the diagnosis.

Your rows should be patients, and your columns should be doctors. You now have a situation that's common in psychometrics (which is why I added the tag).

Then look at the relationships between the scores. Each patient has a mean score, and a score from each doctor. Does the mean score correlate positively with every doctor's score? If not, that doctor is probably not trustworthy (this is called the item-total correlation). Sometimes you remove one doctor from the total score (or mean score) and see if that doctor correlates with the mean of all the other doctors - this is the corrected item total correlation.

You could calculate Cronbach's alpha (which is a form of intra-class correlation), and the alpha without each doctor. Alpha should always rise when you add a doctor, so if it rises when you remove a doctor, that doctor's rating is suspect (this doesn't often tell you anything different from the corrected item-total correlation).

If you use R, this sort of thing is available in the psych package, using the function alpha. If you use Stata, the command is alpha, in SAS it's proc corr, and in SPSS it's under scale, reliability.

Then you can calculate a score, as the mean score from each doctor, or the weighted mean (weighted by the correlation) and see if that score is predictive of the true diagnosis.

Or you could skip that stage, and regress each doctor's score on diagnosis separately, and treat the regression parameters as weights.

Feel free to ask for clarification, and if you want a book, I like Streiner and Norman's "Health Measurement Scales".

-Edit: based on OPs additional info.

Wow, that's a heck of a Cronbach's alpha. The only time I've seen it that high is when a mistake was made.

I would now do logistic regression and look at the ROC curves.

The difference between weighting by regression and correlation depends on how you believe the doctors are responding. Some docs might be generally more confident (without being more skillful), and hence they might use the extreme ranges more. If you want to correct for that, using correlation, rather than regression, does that. I would probably weight by regression, as this keeps the original data (and doesn't discard any information).

Edit (2): I ran logistic regression models in R to see how well each predicted the output. tl/dr: there's nothing between them.

Here's my code:

d <- read.csv("Copy of Cancer data - Weightings.csv")

mrc <- glm(cancer ~ weightrc, data = d, family = "binomial")
mun <- glm(cancer ~ unweight, data = d, family = "binomial")
mca <- glm(cancer ~ weightca, data = d, family = "binomial")
mic <- glm(cancer ~ weightic, data = d, family = "binomial")

d$prc <- predict(mrc, type = "response")
d$pun <- predict(mun, type = "response")
d$pca <- predict(mca, type = "response")
d$pic <- predict(mic, type = "response")

par(mfrow = c(2, 2))
roc(d$cancer, d$prc, ci = TRUE, plot = TRUE)
roc(d$cancer, d$pun, ci = TRUE, plot = TRUE)
roc(d$cancer, d$pca, ci = TRUE, plot = TRUE)
roc(d$cancer, d$pic, ci = TRUE, plot = TRUE)

4 ROC curves

And the output:

> par(mfrow = c(2, 2))
> roc(d$cancer, d$prc, ci = TRUE, plot = TRUE)

roc.default(response = d$cancer, predictor = d$prc, ci = TRUE,     plot = TRUE)

Data: d$prc in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9831
95% CI: 0.9637-1 (DeLong)
> roc(d$cancer, d$pun, ci = TRUE, plot = TRUE)

roc.default(response = d$cancer, predictor = d$pun, ci = TRUE,     plot = TRUE)

Data: d$pun in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9808
95% CI: 0.9602-1 (DeLong)
> roc(d$cancer, d$pca, ci = TRUE, plot = TRUE)

roc.default(response = d$cancer, predictor = d$pca, ci = TRUE,     plot = TRUE)

Data: d$pca in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9854
95% CI: 0.9688-1 (DeLong)
> roc(d$cancer, d$pic, ci = TRUE, plot = TRUE)

roc.default(response = d$cancer, predictor = d$pic, ci = TRUE,     plot = TRUE)

Data: d$pic in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9822
95% CI: 0.9623-1 (DeLong)
  • 1
    $\begingroup$ Very good. And, as your reasoning allows, it's possible some doctor will make alpha worse by contributing unique insight while bucking the trend. $\endgroup$
    – rolando2
    Dec 24, 2016 at 0:39
  • $\begingroup$ @jeremy-miles Thanks for this answer, and the kind offer to field questions about it. I tried to implement what you suggested, and edited the OP to post some of the results. The main thing I'm wondering about is whether I interpreted your post correctly, and also what it would take to show that certain methods of aggregation are working better than other methods in predicting the outcome. $\endgroup$ Dec 24, 2016 at 7:35
  • $\begingroup$ Thanks for posting the data. I'll have a look at it later. (What software are you using?) $\endgroup$ Dec 24, 2016 at 18:09
  • $\begingroup$ @JeremyMiles Thanks for posting this edit! I'm using MATLAB, but I know enough about R to shift up and use that instead, since you've already posted R code. I calculated that Cronbach's Alpha in SPSS - do you get a different value from R? $\endgroup$ Dec 25, 2016 at 12:11
  • 1
    $\begingroup$ Yes, that's what I was thinking. So each doctor gets a different weight. $\endgroup$ Jan 11, 2017 at 4:26

Two out-of-the-box suggestions:

  1. You can use weights on the loss function of your logistic regression, so that the doctor who is very certain that the patient has cancer with P=1 gets double the impact has another who says he has cancer with P=0.75. Do not forget to properly transform your probabilities into weights.
  2. A family of models often neglected are ranking models. Within rankers there are three big groups: listwise, pointwise and pairwise ranking, depending on what your input is. It sounds like you could use pointwise ranking in your case.
  • $\begingroup$ Can you suggest a way to properly transform probabilities into weights? I tried googling this concept but could not find any clear advice on how to do this. $\endgroup$ Dec 26, 2016 at 22:20
  • $\begingroup$ @user1205901, I had in mind something very simple like: Let P=probability of being cancer given by the doctor, then (in python notation): y=[1 if p >= 0.5 else 0 for p in P] and w=[abs(p-0.5)*2 for p in P]. Then train the model: LogisticRegression().fit(X,y,w) $\endgroup$ Dec 26, 2016 at 23:11
  • $\begingroup$ In the loss function, this will give, for example, double the weight to 0.1 than 0.2 for not being cancer (0.1 -> 0.8 and 0.2 -> 0.6). If the doctor is unsure (P~0.5) then the observation contributes almost nothing to the loss. Whatever model he does use needs to support adding a cost vector to the loss function, which most models support. I don't know if this is any good, but it seems trivial to try. He needs to specify a metric first. The loss function can be further tailor to whatever metric he wants to maximize. $\endgroup$ Dec 26, 2016 at 23:18

(This is out of my area of expertise, so the answer by Jeremy Miles may be more reliable.)

Here is one idea.

First, imagine there is no confidence level. Then for each patient $i=1\ldots{N}$, they either have cancer or not $c_i\in\{0,1\}$, and each doctor $j=1\ldots{m}$ either diagnosed them with cancer or not, $d_{ij}\in\{0,1\}$.

A simple approach is to assume that, while the doctors may agree or disagree on a given patient's diagnosis, if we know the patient's true status, then each doctor's diagnosis can be treated as independent. That is, the $d_{ij}$ are conditionally independent given $c_i$. This results in a well defined classifier known as Naive Bayes, with parameters that are easy to estimate.

In particular, the primary parameters are the base rate, $p[c]\approx\tfrac{1}{N}\sum_ic_i$, and the conditional diagnosis likelihoods $$p\big[d_j|c\big]\approx\frac{\sum_id_{ij}c_i}{\sum_ic_i}$$ Note that this latter parameter is a weighted average of the diagnoses for doctor $j$, where the weights are the true patient conditions $c_i$.

Now if this model is reasonable, then one way to incorporate the confidence levels is to adjust the weights. Then the conditional likelihoods would become $$p\big[d_j|c,w_j\big]\approx\frac{\sum_id_{ij}w_{ij}c_i}{\sum_iw_{ij}c_i}$$ Here $w_{ij}\geq{0}$ is a weight that accounts for the confidence level of $d_{ij}$.

Note that if your weights are cast as probabilities $w\in[0,1]$, then you can use the "Bernoulli shortcut" formula $$p\big[d\mid{w}\big]=d^w(1-d)^{1-w}$$ to account for the $d=0$ case appropriately.

Note: This requires that your software give 0^0=1 rather than 0^0=NaN, which is common but worth checking! Alternatively you can ensure $w\in(0,1)$, e.g. if confidence is $k\in\{1\ldots{K}\}$ then $w=k/(K+1)$ would work.

  • $\begingroup$ In the context of the comment by @Wayne : If you say No Cancer (3) = Cancer (2), this is consistent with my weighting model using $w[k]=\frac{k}{K}$, since $\frac{2}{5}=1-\frac{3}{5}$. Alternatively, if you say No Cancer (3) = Cancer (3), this is consistent with $w[k]=\frac{k}{K+1}$, since $\frac{3}{6}=1-\frac{3}{6}$. $\endgroup$
    – GeoMatt22
    Dec 23, 2016 at 18:06
  • $\begingroup$ Can I check that I understand $$p\big[d_j|c,w_j\big]\approx\frac{\sum_id_{ij}w_{ij}c_i}{\sum_iw_{ij}c_i}$$ correctly? If the outcomes are [1,0,1] and a doctor forecasts [0,1,1], and the doctor's weights are [0.2,0.4,0.8], does the weighted conditional diagnosis likelihood come out to be 0.5? $\endgroup$ Dec 25, 2016 at 12:45
  • $\begingroup$ Sorry, I realized I had intended just that the $d$'s be weighted, i.e. $\delta\in[0,1]$, rather than indicators $d\in\{0,1\}$. So for your case $\delta_i=w_i(d_i=1)+(1-w_i)(d_i=0)\implies\delta=[0.8,0.4,0.8]$. Then $p[c,\delta]=\overline{c\delta}=\frac{0.8+0+0.8}{3}=\frac{2}{3}0.8$, while $p[2]=\bar{c}=\frac{2}{3}$ and $p[\delta]=\bar{\delta}=\frac{5}{6}0.8$. So $p[c|\delta]=p[c,\delta]/p[\delta]=0.8$ and $p[\delta|c]=p[c,\delta]/p[c]=0.8$. $\endgroup$
    – GeoMatt22
    Dec 25, 2016 at 18:17

From your question, it appears that what you want to test is your measurement system. In the process engineering realm, this would be an attribute measurement system analysis or MSA.

This link provides some useful information on the sample size needed and the calculations run to conduct a study of this type. https://www.isixsigma.com/tools-templates/measurement-systems-analysis-msa-gage-rr/making-sense-attribute-gage-rr-calculations/

With this study, you would also need the doctor to diagnose the same patient with the same information at least twice.

You can conduct this study one of two ways. You can use the simple cancer/no cancer rating to determine the agreement between physicians and by each physician. Ideally, they should also be able to diagnose with the same level of confidence. You can then use the full 10 point scale to test agreement between and by each physician. (Everyone should agree that cancer (5) is the same rating, that no cancer (1) is the same rating, &c.)

The calculations in the linked website are simple to conduct in any platform you may be using for your tests.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.