My question is not literally the formula for measuring sensitivity and specificity of a test. Instead, I am asking how do different medical tests decide what constitutes a reported TP/TN/FP/FN, and how is that related to the structure of their trial?

For example, I develop a new medical test for condition X which occurs in 10 out of every 100 individuals in the broader population. I claim my test has 95% sensitivity and 99% specificity based on trials. But wouldn't the structure of the trial change the observed accuracy? I could structure it something like:

  1. Test 1,000 random population individuals
  2. Test 1,000 individuals but over-sample those with the condition (maybe 50/50)

I would think scenario 1 would experimentally show my test being less effictive while scenario 2 amplifies it's power. Does the proportion of classes in a study affect how I should infer test accuracy?

Related question would be how do medical sensitivity/specificity numbers change depending on other patient observations? For example:

  1. An a patient walks into a doctor's office and a doctor randomly orders a test with 95%/99% accuracy
  2. A patient showing symptoms consistent with the disease and the doctor orders the same test

The sensitivity is calculated purely from the true cases so the number of the true non-cases is not relevant and vice versa. However if the decision is being made by a human diagnostician and s/he knows the prevalence of the condition it may affect his/her criterion. The positive and negative predictive values are of course strongly affected by the prevalence.

You may be interested in

  author = {Lijmer, J G and Mol, B W and Heisterkamp, S and
     Bonsel, G J and Prins, M H and van der Meulen, J H P
      and Bossuyt, P M M},
  year = {1999},
  title = {Empirical evidence of design-related bias in
     studies of diagnostic tests},
  journal = {Journal of the American Medical Association},
  volume = 282,
  pages = {1061--1066}

which discusses a number of ways in which studies of diagnostic tests can be subject to bias induced by poor design.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.