9
$\begingroup$

How can I calculate the sensitivity and specificity (or analogous measures) of a continuous diagnostic test in predicting a continuous outcome (e.g., blood pressure) without dichotomizing the outcome? Any ideas?

It appears that researchers have done this using mixed effects modeling (see link below), but I'm not familiar with their use of the technique: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3026390/

By the way, I'm most familiar with R, so it would be ideal for the implementation you suggest to be accompanied with an R function (but it's okay if not).

Thank you in advance for any suggestions!

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

As the question is still not answered, here are my 2ct:
I think here are two different topics mixed into this question:

How can I calculate the sensitivity and specificity (or analogous measures) of a continuous diagnostic test in predicting a continuous outcome (e.g., blood pressure) without dichotomizing the outcome?

I take it that you want to measure the performance of the model. The model predicts continuous (metric) outcome from some kind of input (happens to be metric in your example as well, but that doesn't really matter here). This is a regression scenario, not a classification. So you better look for performance measures for regression models, sensitivity and specificity are not what you are looking for*.
Some regression problems have a "natural" grouping into presence and absence of something, which gives a link to classification. For that you may have a bimodal distribution: lots of cases with absence, and a metric distribution of values for cases of presence. For example, think of a substance that contaminates some product. Many of the product samples will not contain the contaminant, but for those that do, a range of concentrations is observed.

However, this is not the case for your example of blood pressure (absence of blood pressure is not a sensible concept here). I'd even guess that blood pressures come in a unimodal distribution. All that points to a regression problem without close link to classification.

* With the caveat that both words are used in analytical chemistry for regression (calibration), but with a different meaning: there, the sensitivity is the slope of the calibration/regression function, and specific sometimes means that the method is completely selective, that is it is insensitive to other substances than the analyte, and no cross-sensitivities occur.
A. D. McNaught und A. Wilkinson, eds.: Compendium of Chemical Terminology (the “Gold Book”). Blackwell Scientific, 1997. ISBN: 0-9678550-9-8. DOI: doi:10.1351/ goldbook. URL: http://goldbook.iupac.org/.

Analogues of sensitivity and specificity for continuous outcomes

On the other hand, if the underlying nature of the problem is a classification, you may nevertheless find yourself describing it better by a regression:

  • the regression describes a degree of belonging to the classes (as in fuzzy sets).
  • the regression models (posterior) probability of beloning to the classes (as in logistic regression)
  • your cases can be described as mixtures of the pure classes (very close to "normal" regression, the contamination example above)

For these cases it does make sense to extend the concepts behind sensitivity and specificity to "continuous outcome classifiers". The basic idea is to weight each case according to its degree of belonging to the class in question. For sensitivity and specificity that refers to the reference label, for the predictive values to the predicted class memberships. It turns out that this leads to a very close link to regression-type performance measures.

We recently described this in C. Beleites, R. Salzer and V. Sergo:
Validation of Soft Classification Models using Partial Class Memberships: An Extended Concept of Sensitivity & Co. applied to Grading of Astrocytoma Tissues
Chemom. Intell. Lab. Syst., 122 (2013), 12 - 22.
The link points to the home page of the R package implementing the proposed perfromance measures.

Again, the blood pressure example IMHO is not adequately described as classification problem. However, you may still want to read the paper - I think the formulation of the reference values there will make clear that blood pressure is not sensibly described in a way that is suitable for classification.

(If you formulate a continuous degree of "high blood pressure" that would itself be a model, and a different one from the problem you describe.)

I had only a quick glance at the paper you linked, but if I understood correctly the authors use thresholds (dichotomize) for both modeling strategies: for the continuous prediction is further processed: a prediction interval is calculated and compared to some threshold. In the end, they have a dichotomous prediction, and generate the ROC by varying the specification for the interval.
As you specify that you want to avoid this, the paper doesn't seem to be overly relevant.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Sounds as if the methods in that paper are highly problematic. The original question was not answered because the question implies the use of questionable methodology. $\endgroup$
    – Frank Harrell
    Commented Jul 31, 2013 at 11:50
  • $\begingroup$ @FrankHarrell: yes, at least the naming of the method in the paper is misleading as the final prediction is not continuous. But I understood the mentioning of the paper as showing that the OP did look into literature, and thought that the paper may help - which it doesn't. $\endgroup$
    – cbeleites
    Commented Jul 31, 2013 at 18:53
2
$\begingroup$

Trying to do this with continuous variables will expose the severe problems with backwards time-order measures even in the binary case (i.e., predicting X from Y in general).

Improve this answer
$\endgroup$
8
  • $\begingroup$ It doesn't have to be with backwards time-order measures. I'm just looking for an analogue of sensitivity and specificity in the case where the outcome or dependent variable is continuous. $\endgroup$
    – itpetersen
    Commented Feb 1, 2013 at 14:03
  • $\begingroup$ The analog would have to be backwards, i.e., involve the distribution of $X$ given $Y$. Otherwise it is not analogous. $\endgroup$
    – Frank Harrell
    Commented Feb 1, 2013 at 14:13
  • 2
    $\begingroup$ You are right; you have stated a very reasonable goal that is in the correct time order. The sensitivity analog would be to predict prenatal testosterone exposure from future degree of aggression. To answer your question I would use a generalized $R^2$ measure, generalized ROC area (i.e. Somers' $D_{xy}$ rank correlation between predicted and observed), and the histogram of predicted aggression degree - the larger the variety the more discrimination the predictions. $\endgroup$
    – Frank Harrell
    Commented Feb 1, 2013 at 16:11
  • 2
    $\begingroup$ That's the reason for generalized in what I wrote. $D_{xy}$ is proportional to generalized ROC area which reduces to ordinary ROC area if $Y$ is binary. It is an unfortunate accident of nature that concordance probability can be derived from sensitivity and specificity. Just think of the more general (and non-backward-looking) concordance probability between $X$ and $Y$ ($c$-index; equals ROC area in binary $Y$ case). Concordance probability (easily described for continuous $Y$) or ROC area can be used without creating an ROC curve at all. No need to predict the past from the future. $\endgroup$
    – Frank Harrell
    Commented Feb 1, 2013 at 16:59
  • 2
    $\begingroup$ My R Hmisc and rms packages handle this. In Hmisc see the rcorr.cens function. $\endgroup$
    – Frank Harrell
    Commented Feb 1, 2013 at 20:59
0
$\begingroup$

Taken loosely, sensitivity means the ability to respond to something if it's present, and specificity means the ability to suppress responding when it's absent. For continuous variables, sensitivity corresponds to the slope of the regression of the obtained measures on the true values of the variable being measured, and specificity corresponds to the standard error of measurement (i.e., the standard deviation of the obtained measures when the quantity being measured does not vary).

EDIT, responding to comments by Frank Harrell and cbeleites. I was trying to give conceptual analogs of sensitivity and specificity. For continuous variables, the basic idea of sensitivity is that if two objects (or the same object at different times or under different conditions, etc) differ on the variable we are trying to measure, then our obtained measures should also differ, with bigger true differences leading to bigger measured differences.

The regression of any variable, say $Y$, on any other, say $X$, is simply the conditional expected value, $\mathrm{E}\,Y|X$, treated as a function of $X$. The sensitivity of $Y$ to $X$ is the slope of that function -- i.e., its derivative with respect to $X$ -- evaluated at whatever values of $X$ are of interest, and possibly averaged with weights that reflect the relative importance or frequency of occurrence of different $X$-values.

The basic idea of specificity is the converse of sensitivity: if $Y$ has high specificity and there are no true differences on $X$ then all our measured $Y$-values should be the same, regardless of whatever differences there may be among the objects on variables other than $X$; $Y$ should not respond to those differences. When $X$ is constant, higher variability among the $Y$-values implies lower specificity. The conditional standard deviation -- i.e., the s.d. of $Y|X$, again treated as a function of $X$ -- is an inverse measure of specificity. The ratio of the conditional slope over the conditional s.d. is a signal-to-noise ratio, and its square is referred to in psychometrics as the information function.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That does not sound quite right. Specificity does not involve variances and sens. and spec. only apply to binary quantities. $\endgroup$
    – Frank Harrell
    Commented Jul 31, 2013 at 17:05
  • $\begingroup$ @Ray: the meaning/definition of both terms for regression (chemical calibration) is so different from their meaning in classification that this should IMHO more emphasized. I've never heard about your definition of specificity, though. Can you give a reference? $\endgroup$
    – cbeleites
    Commented Jul 31, 2013 at 18:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.