How should I approach this binary prediction problem?

Question

I've got a dataset with the following format.

There's a binary outcome cancer/no cancer. Every doctor in the dataset has seen every patient and given an independent judgment on whether the patient has cancer or not. The doctors then give their confidence level out of 5 that their diagnosis is correct, and the confidence level is displayed in the brackets.

I've tried various ways to get good forecasts out of this dataset.

It works pretty well for me to just average across the doctors, ignoring their confidence levels. In the table above that would have produce correct diagnoses for Patient 1 and Patient 2, although it would have incorrectly said that Patient 3 has cancer, since by a 2-1 majority the doctors think Patient 3 has cancer.

I also tried a method in which we randomly sample two doctors, and if they disagree with each other then the deciding vote goes to whichever doctor is more confident. That method is economical in that we don't need to consult a lot of doctors, but it also increases the error rate quite a bit.

I tried a related method in which we randomly select two doctors, and if they disagree with each other we randomly select two more. If one diagnosis is ahead by at least two 'votes' then we resolve things in favor of that diagnosis. If not, we keep sampling more doctors. This method is pretty economical and doesn't make too many mistakes.

I can't help feeling that I'm missing some more sophisticated way of doing things. For instance, I wonder if there is some way I could divide the dataset into training and test sets, and work out some optimal way to combine the diagnoses, and then see how those weights perform on the test set. One possibility is some sort of method that lets me downweight doctors who kept making mistakes on the trial set, and perhaps upweight diagnoses that are made with high confidence (confidence does correlate with accuracy in this dataset).

I've got various datasets matching this general description, so the sample sizes vary and not all the datasets relate to doctors/patients. However, in this particular dataset there are 40 doctors, who each saw 108 patients.

EDIT: Here is a link to some of the weightings that result from my reading of @jeremy-miles's answer.

1. Unweighted results are in the first column. Actually in this dataset the maximum confidence value was 4, not 5 as I mistakenly said earlier. Thus following @jeremy-miles's approach the highest unweighted score any patient could get would be 7. That would mean that literally every doctor asserted with a confidence level of 4 that that patient had cancer. The lowest unweighted score any patient could get is 0, which would mean that every doctor asserted with a confidence level of 4 that that patient did not have cancer.

2. Weighting by Cronbach's Alpha. I found in SPSS that there was an overall Cronbach's Alpha of 0.9807. I tried to verify that this value was correct by calculating Cronbach's Alpha in a more manual way. I created a covariance matrix of all 40 doctors, which I paste here. Then based on my understanding of the Cronbach's Alpha formula $\alpha = \frac{K}{K-1}\left(1-\frac{\sum \sigma^2_{x_i}}{\sigma^2_T}\right)$ where $K$ is the number of items (here the doctors are the 'items') I calculated $\sum \sigma^2_{x_i}$ by summing all the diagonal elements in the covariance matrix, and $\sigma^2_T$ by summing all the elements in the covariance matrix. I then got $\alpha = \frac{40}{40-1}\left(1-\frac{8.7915}{200.7112}\right)=0.9807$ I then calculated the 40 different Cronbach Alpha results that would occur when each doctor got removed from the dataset. I weighted any doctor who contributed negatively to Cronbach's Alpha at zero. I came up with weights for the remaining doctors proportional to their positive contribution to Cronbach's Alpha.

3. Weighting by Total Item Correlations. I calculate all the Total Item Correlations, and then weight each doctor proportional to the size of their correlation.

4. Weighting by Regression Coefficients.

One thing I'm still not sure about is how to say which method is working "better" than the other. Previously I had been calculating things like the Peirce Skill Score, which is appropriate for instances in which there is a binary prediction and a binary outcome. However, now I have forecasts ranging from 0 to 7 instead of 0 to 1. Should I convert all the weighted scores > 3.50 to 1, and all the weighted scores < 3.50 to 0?

Answer 1

First, I would see if the doctors agree with each other. You can't analyze 50 doctors separately, because you'll overfit the model - one doctor will look great, by chance.

You might try to combine confidence and diagnosis into a 10 point scale. If a doctors says that the patient doesn't have cancer, and they are very confident, that's a 0. If the doc says they do have cancer and they are very confident, that's a 9. If they doc says they don't, and are not confident, that's a 5, etc.

When you're trying to predict, you do some sort of regression analysis, but thinking about the causal ordering of these variables, it's the other way around. Whether the patient has cancer is the cause of the diagnosis, the outcome is the diagnosis.

Your rows should be patients, and your columns should be doctors. You now have a situation that's common in psychometrics (which is why I added the tag).

Then look at the relationships between the scores. Each patient has a mean score, and a score from each doctor. Does the mean score correlate positively with every doctor's score? If not, that doctor is probably not trustworthy (this is called the item-total correlation). Sometimes you remove one doctor from the total score (or mean score) and see if that doctor correlates with the mean of all the other doctors - this is the corrected item total correlation.

You could calculate Cronbach's alpha (which is a form of intra-class correlation), and the alpha without each doctor. Alpha should always rise when you add a doctor, so if it rises when you remove a doctor, that doctor's rating is suspect (this doesn't often tell you anything different from the corrected item-total correlation).

If you use R, this sort of thing is available in the psych package, using the function alpha. If you use Stata, the command is alpha, in SAS it's proc corr, and in SPSS it's under scale, reliability.

Then you can calculate a score, as the mean score from each doctor, or the weighted mean (weighted by the correlation) and see if that score is predictive of the true diagnosis.

Or you could skip that stage, and regress each doctor's score on diagnosis separately, and treat the regression parameters as weights.

Feel free to ask for clarification, and if you want a book, I like Streiner and Norman's "Health Measurement Scales".

-Edit: based on OPs additional info.

Wow, that's a heck of a Cronbach's alpha. The only time I've seen it that high is when a mistake was made.

I would now do logistic regression and look at the ROC curves.

The difference between weighting by regression and correlation depends on how you believe the doctors are responding. Some docs might be generally more confident (without being more skillful), and hence they might use the extreme ranges more. If you want to correct for that, using correlation, rather than regression, does that. I would probably weight by regression, as this keeps the original data (and doesn't discard any information).

Edit (2): I ran logistic regression models in R to see how well each predicted the output. tl/dr: there's nothing between them.

Here's my code:

d <- read.csv("Copy of Cancer data - Weightings.csv")

mrc <- glm(cancer ~ weightrc, data = d, family = "binomial")
mun <- glm(cancer ~ unweight, data = d, family = "binomial")
mca <- glm(cancer ~ weightca, data = d, family = "binomial")
mic <- glm(cancer ~ weightic, data = d, family = "binomial")

d$prc <- predict(mrc, type = "response")
d$pun <- predict(mun, type = "response")
d$pca <- predict(mca, type = "response")
d$pic <- predict(mic, type = "response")

par(mfrow = c(2, 2))
roc(d$cancer, d$prc, ci = TRUE, plot = TRUE)
roc(d$cancer, d$pun, ci = TRUE, plot = TRUE)
roc(d$cancer, d$pca, ci = TRUE, plot = TRUE)
roc(d$cancer, d$pic, ci = TRUE, plot = TRUE)

And the output:

> par(mfrow = c(2, 2))
> roc(d$cancer, d$prc, ci = TRUE, plot = TRUE)

Call:
roc.default(response = d$cancer, predictor = d$prc, ci = TRUE,     plot = TRUE)

Data: d$prc in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9831
95% CI: 0.9637-1 (DeLong)
> roc(d$cancer, d$pun, ci = TRUE, plot = TRUE)

Call:
roc.default(response = d$cancer, predictor = d$pun, ci = TRUE,     plot = TRUE)

Data: d$pun in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9808
95% CI: 0.9602-1 (DeLong)
> roc(d$cancer, d$pca, ci = TRUE, plot = TRUE)

Call:
roc.default(response = d$cancer, predictor = d$pca, ci = TRUE,     plot = TRUE)

Data: d$pca in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9854
95% CI: 0.9688-1 (DeLong)
> roc(d$cancer, d$pic, ci = TRUE, plot = TRUE)

Call:
roc.default(response = d$cancer, predictor = d$pic, ci = TRUE,     plot = TRUE)

Data: d$pic in 81 controls (d$cancer 0) < 27 cases (d$cancer 1).
Area under the curve: 0.9822
95% CI: 0.9623-1 (DeLong)

Answer 2

Two out-of-the-box suggestions:

1. You can use weights on the loss function of your logistic regression, so that the doctor who is very certain that the patient has cancer with P=1 gets double the impact has another who says he has cancer with P=0.75. Do not forget to properly transform your probabilities into weights.
2. A family of models often neglected are ranking models. Within rankers there are three big groups: listwise, pointwise and pairwise ranking, depending on what your input is. It sounds like you could use pointwise ranking in your case.

Answer 3

(This is out of my area of expertise, so the answer by Jeremy Miles may be more reliable.)

Here is one idea.

First, imagine there is no confidence level. Then for each patient $i=1\ldots{N}$, they either have cancer or not $c_i\in\{0,1\}$, and each doctor $j=1\ldots{m}$ either diagnosed them with cancer or not, $d_{ij}\in\{0,1\}$.

A simple approach is to assume that, while the doctors may agree or disagree on a given patient's diagnosis, if we know the patient's true status, then each doctor's diagnosis can be treated as independent. That is, the $d_{ij}$ are conditionally independent given $c_i$. This results in a well defined classifier known as Naive Bayes, with parameters that are easy to estimate.

In particular, the primary parameters are the base rate, $p[c]\approx\tfrac{1}{N}\sum_ic_i$, and the conditional diagnosis likelihoods $$p\big[d_j|c\big]\approx\frac{\sum_id_{ij}c_i}{\sum_ic_i}$$ Note that this latter parameter is a weighted average of the diagnoses for doctor $j$, where the weights are the true patient conditions $c_i$.

Now if this model is reasonable, then one way to incorporate the confidence levels is to adjust the weights. Then the conditional likelihoods would become $$p\big[d_j|c,w_j\big]\approx\frac{\sum_id_{ij}w_{ij}c_i}{\sum_iw_{ij}c_i}$$ Here $w_{ij}\geq{0}$ is a weight that accounts for the confidence level of $d_{ij}$.

Note that if your weights are cast as probabilities $w\in[0,1]$, then you can use the "Bernoulli shortcut" formula $$p\big[d\mid{w}\big]=d^w(1-d)^{1-w}$$ to account for the $d=0$ case appropriately.

Note: This requires that your software give 0^0=1 rather than 0^0=NaN, which is common but worth checking! Alternatively you can ensure $w\in(0,1)$, e.g. if confidence is $k\in\{1\ldots{K}\}$ then $w=k/(K+1)$ would work.

Answer 4

From your question, it appears that what you want to test is your measurement system. In the process engineering realm, this would be an attribute measurement system analysis or MSA.

This link provides some useful information on the sample size needed and the calculations run to conduct a study of this type. https://www.isixsigma.com/tools-templates/measurement-systems-analysis-msa-gage-rr/making-sense-attribute-gage-rr-calculations/

With this study, you would also need the doctor to diagnose the same patient with the same information at least twice.

You can conduct this study one of two ways. You can use the simple cancer/no cancer rating to determine the agreement between physicians and by each physician. Ideally, they should also be able to diagnose with the same level of confidence. You can then use the full 10 point scale to test agreement between and by each physician. (Everyone should agree that cancer (5) is the same rating, that no cancer (1) is the same rating, &c.)

The calculations in the linked website are simple to conduct in any platform you may be using for your tests.