I am writing a supervised classifier for a particular condition. I have two sets of data for my "gold standard", against which I will test my classifier:

  • a Positive set, in which all samples have the condition;
  • a Negative set, in which none have the condition.

For illustration purposes, here are the results of my classifier, aka the confusion matrix:

| TP=354 | FP= 20 |
| FN= 69 | TN=404 |

That gives Sensitivity=84% and Specificity=95%

The problem

I am not sure that the Gold Standard data has itself 100% sensitivity and 100% specificity. I estimate 95% sensitivity for the positive condition, and 90% specificity for the negative condition.

How can I adjust my results? Or how to communicate this uncertainty?



This should be relatively straightforward given one very crucial assumption, that we will get to. Let's establish some notation. Let's define $X$ to be the random variable obtained by randomly selecting a data point from your set and classifying it using your classifier. $Y$ as the random variable obtained by randomly selecting a data point from your set and getting it's gold standard class label. And $Z$ as the random variable obtained by randomly selecting a data point from your set and getting its true label.

Now let's summarize the information we have so far. The things we know or believe we know are

$$ P(X|Y=1), P(X|Y=0), P(Y|Z=1), P(Y|Z=0). $$ These are given by the sensitivity and specificity values that you have measured or assumed. So more succinctly we know: $$ P(X|Y), P(Y|Z) $$ What we want to know is $P(X|Z)$, the true sensitivity and specificity of your classifier. We can obtain this from $P(X,Y|Z)$ by summing over all (both) possible values of $Y$, if we can get $P(X,Y|Z)$. It is a simple consequence of the definition of conditional probability that $$ P(X,Y|Z) = P(X|Y,Z) \cdot P(Y|Z), $$ [if this is new to you remove the Z and it will be quite familiar]. But we don't know $P(X|Y,Z)$. Therefore the pivotal assumption, without which I don't think we can do anything (unless you know the true label of individuals in which case you should train on that), is that $X,Z$ are conditionally independent, given $Y$, in which case $P(X|Y,Z) = P(X|Y)$, that is the only way the true label effects your prediction is by effecting the (gold) label you used to train your predictor.

So if your comfortable with that assumption, then we can proceed to calculate: $$ P(X=1|Z=1) = P(X=1|Y=1)P(Y=1|Z=1) + P(X=1|Y=0)P(Y=0|Z=1) = 0.84\cdot0.95 + 0.05\cdot0.05 $$

I will leave the other calculation to you. Hope this was helpful!

  • 2
    $\begingroup$ Just an addendum: the condition $P(X|Y,Z) = P(X|Y)$ means that the classifier, which is a function of observed positive, performs just as well on the truly negative observations that are labeled "positive" as on the correctly labeled observations. That is, the process by which the data is mislabeled does not affect your classifier. This could be a non-trivial assumption if the $Y$-$Z$ mislabeling itself is non-random with respect to one of your classifier inputs. $\endgroup$ – shadowtalker Mar 23 '15 at 16:00
  • $\begingroup$ I couldn't agree more. That being said, from my understanding of the set-up of the problem, I don't believe we can do better. OP, could you give us an idea for how you arrived at your beliefs regarding the sensitivity and specificity of the original labels? (A few days later, some of the people who ate "non-poisonous" mushrooms died, but that could have been from any number of causes) $\endgroup$ – jlimahaverford Mar 23 '15 at 22:01
  • $\begingroup$ Thank you for your answers. I was trying to avoid domain specifics, but since you asked, here are my beliefs: the data is related to genetics. The True Positive set comes from a database HGMD while the True Negative set is based on an assumption (which is, MAF>=0.20 is a benign variant). I have heard in conferences that the HGMD contained "errors" (but I cannot find citations nor proper quantification); while the benign assumption does not hold at all times. $\endgroup$ – Hugues Fontenelle Mar 24 '15 at 8:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.