How do I compute a cutoff based on sensitivity/specificity when the characteristics of my sample is different from the population?

Question

I have a dataset containing the performance of a novel instrument to screen for disease A. The novel instrument uses a scoring system to score the subject to determine if they have disease A. I then proceed to use logistic regression to use the novel instrument and demographic variables such as age and gender to classify them into either having or not having disease A and comparing it with the gold standard result.

Since the prevalence of disease A is different in my sampling and in the actual population, how should I handle this problem? I see a couple of possible solutions:

1. Conduct the logistic regression and use the predicted values for each subject to compute for the sensitivity and specificity of the regression.
2. Conduct the logistic regression in the sample. Sample from my existing sample to create a new sample that corresponds to the population proportion of people with and without disease A. Perform the prediction to compute for the sensitivity and specificity of the regression.

My questions are:

• Should I adopt either strategy (1) or (2)?
• During my search for answers, I came across: Case weighted logistic regression. Following the thread on the mailing list, I see a way to make replicate weights for each subject (via survey package). Why should the proportion (in my case) of people with and without disease A affect the results of the logistic regression? Is the replicate weights method a valid way of overcoming this issue or would strategy (2) work as well?

Answer 1

I don't think you actually have a problem due to the sampling proportions in the disease/no disease dimension.

Different prevalence of disease in your sample and the population isn't an issue for calculating sensitivity/specificity -- these two numbers just ask (1 -- sensitivity) whether the tool works well for detecting disease amongst those with disease, and (2 -- specificity) whether it works well correctly determining "no disease" amongst those without the disease. If you think about these two points (see Wikipedia http://en.wikipedia.org/wiki/Sensitivity_and_specificity for formulae), you'll see that the calculations are independent of disease prevalence.

Your sampling scenario is however an issue for calculating positive predictive value and negative predictive value -- but one could factor the known population prevalence into a calculation to combine this information with your sensitivity/specificity in order to estimate PPV/NPV. Again, Wikipedia covers this issue (http://en.wikipedia.org/wiki/Positive_predictive_value#Problems_with_positive_predictive_value which also links through to a Statistics Notes one-pager from the BMJ that touches on this issue briefly: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2540558/)

A more general point to consider for your overall approach -- if you use logistic regression to determine your sensitivity and specificity (really, one would be calculating an area under the curve equivalent using the c-statistic) then you are actually measuring sensitivity and specificity of the entire model -- that is, sensitivity/specificity of your new diagnostic test alongside the information in age/gender and the other covariates being specified.