1
$\begingroup$

I know when a new measurement technique is developed, it's sensitivity and specificity is measured against a "gold standard."

I'm wondering what the gold standard itself is measured against? How was its specificity and sensitivity determined? How does one decide to replace one "gold standard" with another that might have higher specificity and sensitivity?

$\endgroup$

3 Answers 3

3
$\begingroup$

My expertise is in DNA genomics, so I'll use it.

Creating a gold standard reference is critical in genome sequencing. While comparing an experiment to a gold standard is simple, creating the gold standard itself is a very difficult job. My institution has a team of world-class scientists for just doing that. They would use domain knowledge (e.g biology and bioinformatics) to verify the reference. It's a highly technical process, and has nothing to do with statistics.

The gold standard reference is considered a set of perfect true-positives. There is no need to calculate sensitivity and specificity. Everything is measured relative to the standard. If you want to change the standard, you'd have to convince the scientists, and it's not so easy.

$\endgroup$
1
$\begingroup$

The gold standard is suppose to be the true - by definition its specificity and sensitivity should be prefect.

In many cases, is easy to create a gold standard in such a level. For example, there are cases in which finding the correct label is hard at a given point of time but easy after a certain period as in loan return and customer churn. Other cases are when it is easy to judge each case but hard to define a rule (a classifier) like in judging if there is a cat in a picture.

Not that in some cases the definition itself can be vague and the gold standard can be used to enforce interpretation (e.g., is a lion considered a cat? and a leopard?)

There are cases in which building a gold standard that is totally reliable is hard. It is common when using crowd sourcing for labelling. The first good news that you can aggregate the labels while considering the precision and recall of each labeler using a Dawid-Skene estimator Maximum likelihood estimation of observer error-rates using the EM algorithm (1979)

The other good news are that you can learn in the presence of white noise (Learning from noisy examples) and to some degree even with malicious errors (Learning in the presence of malicious errors) so even if your gold standard is not perfect you might still be able to learn well.

$\endgroup$
3
  • $\begingroup$ I appreciate your bottom two paragraphs, but your definition of "gold standard" refers to the theorized ideal, but not "gold standard" is used in practice. An MRA is the gold standard for diagnosing an aortic dissection, but only has a sensitivity of 95% and specificity of 92%. My question is: how is the sensitivity and specificity of an MRA decided upon if there's no "ideal standard" to compare it against? SmallChess answer gives some indication -- scientist judgment. But I assume there's a clearer way of calculating this? $\endgroup$ Apr 14, 2017 at 17:42
  • $\begingroup$ Indeed, there are cases in which it is hard to determine the label of a concept. Note that in order to know the sensitivity and specificity of the gold standard, you should be able the determine the label. I'm not familiar with MRA. A possible approach could be using a small set of patients and label them for aortic dissection (e.g., post mortem). You measure the MRA itself against that gold standard. Once you have that, you use the MRA as the label for other techniques (since MRA data is much more common and easy to use). $\endgroup$
    – DaL
    Apr 18, 2017 at 9:53
  • $\begingroup$ @SmallChess is right saying that in some cases the labelling effort is hard and requires a lot of knowledge. A possible machine learning approach is to give up the hard cases. You can do it by adding another attribute of hardness to the labelling process. Then you can training and test your classifier with the hard part removed. If the hard part is not too large or of significant importance, that could help you. After all, most cats classifiers are indifferent to the classification of a lion. $\endgroup$
    – DaL
    Apr 18, 2017 at 11:18
0
$\begingroup$

I know when a new measurement technique is developed, it's sensitivity and specificity is measured against a "gold standard

No, not quite/not necessarily. I think you and I have a different definition of gold standard, so we're framing the matter a little differently from one another. It doesn't really matter, as the two positions will be reconciled by the end of the answer.

The Cliff's Notes:

The sensitivity and specificity of a measure is calculated by comparing it against the "true state of affairs". If this is unknowable or too onerous for widespread use, then there will be an equivalent operational definition that one works with.

Take, for example, measurements of time (they too can be converted to sensitivity/specificity). The definition of 1 second is the distance travelled by light in a vacuum in 1/299792458 second.

I presume that you view this as the gold standard measure of time, whereas I would define it as the objective standard of time. Instead, I frame the gold standard of time as an atomic clock.

I'm wondering what the gold standard itself is measured against? How was its specificity and sensitivity determined? How does one decide to replace one "gold standard" with another that might have higher specificity and sensitivity?

The above example of time is useful in getting your head this issue.

It's just not possible for an atomic clock to be more precise than the working definition of time itself.

That said, under some circumstances, what you're saying makes far more sense, both conceptually and intuitively. This is particularly a relevant if there is no objective marker to speak of, per se.

A good example of this problem is with psychiatric disorders. A questionnaire that measures, say social anxiety disorder may have a sensitivity and specificity of 0.75 and 0.80, but this is problematic when its predictive utility is based upon the "objective" diagnostic standard of a clinical interview… which itself is susceptible to lack of interrater reliability. In some sense what you're measuring is the shadow of a shadow.

How does one decide to replace one "gold standard" with another that might have higher specificity and sensitivity?

In your definition of gold standard measures, think of the gold standard as being a representation of an objective measure. Here, the gold standard cannot be replaced without obliging us to redefine what it is that is being measured in the first place.

$\endgroup$
6
  • $\begingroup$ Thanks for your clear response; however, my question was referring to situations for which there is no "objective standard" -- let's say we are trying to measure whether someone has prostate cancer. The gold standard measurement is a biopsy under a microscope. However, biopsy itself has sensitivity and specificity values in the medical literature. The question is what other method were they using that was somehow more accurate than the best available method for assessing prostate cancer? $\endgroup$ Oct 30, 2017 at 14:34
  • $\begingroup$ @Parseltongue here, the biopsy is a small tissue sample of the entire prostate, and so only examines a small percentage of the entire prostate volume (particularly when the prostate is enlarged). the "objective standard" would be for an experienced pathologist to examine the histology of the entire prostate. $\endgroup$
    – faustus
    Oct 30, 2017 at 17:05
  • $\begingroup$ Correct, but the point is that there are error bars all the way down. The pathologist himself does not have a sensitivity and specificity of 100%. He is using a microscope which has error bars, he may be distracted, he may miss a piece of infected tissue, and so on. My question is -- how are the sensitivity and specificity of these procedures examined? How can we know how successful the "gold standard" is in operationalizing the "objective definition" of whatever is in question. $\endgroup$ Oct 31, 2017 at 17:32
  • $\begingroup$ The answer IS NOT to just assume the pathologist examining the entire prostate has an accuracy of 100%, otherwise how would we know if some better instrument (like an AI) came around that was better than the pathologist? What outside external truth would we be comparing the pathologist and the AI's performance against if the thing in question is itself ambiguously defined by imperfect measurements. $\endgroup$ Oct 31, 2017 at 17:36
  • $\begingroup$ LOL then you would use consensus from multiple pathologists as the benchmark. the problem is that the way you're viewing this renders every sensitivity/specifity value inaccurate. while this is correct, it's not entirely helpful to take it to that extreme. $\endgroup$
    – faustus
    Nov 1, 2017 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.