Correcting naïve Sensitivity and Specificity for classifier tested against imperfect gold standard

Question

I am writing a supervised classifier for a particular condition. I have two sets of data for my "gold standard", against which I will test my classifier:

• a Positive set, in which all samples have the condition;
• a Negative set, in which none have the condition.

For illustration purposes, here are the results of my classifier, aka the confusion matrix:

| TP=354 | FP= 20 |
|--------|--------|
| FN= 69 | TN=404 |

That gives Sensitivity=84% and Specificity=95%

The problem

I am not sure that the Gold Standard data has itself 100% sensitivity and 100% specificity. I estimate 95% sensitivity for the positive condition, and 90% specificity for the negative condition.

How can I adjust my results? Or how to communicate this uncertainty?

Answer 1

Hugues,

This should be relatively straightforward given one very crucial assumption, that we will get to. Let's establish some notation. Let's define $X$ to be the random variable obtained by randomly selecting a data point from your set and classifying it using your classifier. $Y$ as the random variable obtained by randomly selecting a data point from your set and getting it's gold standard class label. And $Z$ as the random variable obtained by randomly selecting a data point from your set and getting its true label.

Now let's summarize the information we have so far. The things we know or believe we know are

$$ P(X|Y=1), P(X|Y=0), P(Y|Z=1), P(Y|Z=0). $$ These are given by the sensitivity and specificity values that you have measured or assumed. So more succinctly we know: $$ P(X|Y), P(Y|Z) $$ What we want to know is $P(X|Z)$, the true sensitivity and specificity of your classifier. We can obtain this from $P(X,Y|Z)$ by summing over all (both) possible values of $Y$, if we can get $P(X,Y|Z)$. It is a simple consequence of the definition of conditional probability that $$ P(X,Y|Z) = P(X|Y,Z) \cdot P(Y|Z), $$ [if this is new to you remove the Z and it will be quite familiar]. But we don't know $P(X|Y,Z)$. Therefore the pivotal assumption, without which I don't think we can do anything (unless you know the true label of individuals in which case you should train on that), is that $X,Z$ are conditionally independent, given $Y$, in which case $P(X|Y,Z) = P(X|Y)$, that is the only way the true label effects your prediction is by effecting the (gold) label you used to train your predictor.

So if your comfortable with that assumption, then we can proceed to calculate: $$ P(X=1|Z=1) = P(X=1|Y=1)P(Y=1|Z=1) + P(X=1|Y=0)P(Y=0|Z=1) = 0.84\cdot0.95 + 0.05\cdot0.05 $$

I will leave the other calculation to you. Hope this was helpful!