2
$\begingroup$

I'm interested in notation and terminology regarding performance measures related to Sensitivity (SEN) and Specificity (SPC). These two measures are commonly used in binary classification problems. I'm interested in analogous measures for probabilistic classification.

$\textbf{Standard Scenario (Binary Classification):}$

Let $Y$ be a binary response variable where $Y=0$ is a "negative" result and $Y=1$ is a "positive" result. Let $X\in\{0,1\}$ be a test aimed at predicting $Y$. Then, it can be shown that $$\mathrm{SEN} = \dfrac {E[XY]} {E[Y]}$$ $$\mathrm{SPC} = \dfrac {E[(1-X)(1-Y)]} {E[1-Y]}$$ *I realize these measures are typically defined with different notation using empirical counts rather than as expectations of random variables.

$\textbf{Generalized Scenario (Probabilistic Classification):}$

Let $Y$ be the same as before but now $X\in[0,1]$ is a continuous random variable such that $P(Y=1 \mid X=x)=x$. Certainly, $\mathrm{SEN}$ and $\mathrm{SPC}$ - as defined above - are useful measures of the quality of the predictor, $X$.

$\textbf{Questions:}$

  1. Is it acceptable to refer to these quantities as the Sensitivity and Specificity of $X$ in the generalized case? Or are different names and considerations warranted when $X$ is a probabilistic classifier?
  2. Are there any references to this (or a related) concept in the academic literature? I'm struggling to find any.
$\endgroup$
1
$\begingroup$

Thinking in terms of decision theory can help you make continuous analogues out of the quantities of sensitivity and specificity.

Bishop pp. 40 illustrates a binary classification problem along a single (continuous) input dimension, using a stylized representation of joint probabilities $p(c,C_k)$ for each of 2 classes:

enter image description here

If you are classifying a continuous $x$ into one of two classes $C_1, C_2$ then you are effectively drawing a decision boundary/surface $\hat{x}$ in such a way that the input space for $x$ is divided into decision regions $R_1, R_2$. The true decision boundary $x_0$ is unknown.

The analogous terms to "true positive" and "true negative" would be the probability of correctly classifying a point in either $C_1$ or $C_2$, depending on what those $C_k$ actually represent. In the chart above, they are represented by white regions under the curve. To evaluate them,

  • The probability of correctly classifying a point in $C_1$ is $\int_{R_1} p(x,C_1)dx$
  • The probability of correctly classifying a point in $C_2$ is $\int_{R_2} p(x,C_2)dx$

You could also calculate the related "false positive" and "false negative" quantities, which are represented by the coloured red, green & blue areas under the curve:

  • The probability of incorrectly classifying a point in $C_1$ as $C_2$ is $\int_{R_2} p(x,C_1)dx$
  • The probability of incorrectly classifying a point in $C_2$ as $C_1$ is $\int_{R_1} p(x,C_2)dx$

Then sensitivity and specificity would simply be the appropriate ratios of those regions, based on which of $C_1,C_2$ represents the "positive" and which represents the "negative":

$\text{sensitivity} = \frac{\text{true positives}}{\text{true positives} + \text{false negatives}}$

$\text{specificity} = \frac{\text{true negatives}}{\text{true negatives} + \text{false positives}}$

This isn't just a theoretical observation. In our example, the green & blue areas under the curve cannot be reduced, but it should be possible to minimise the red area by minimising the expected loss. This actually solves for the optimal decision boundary/surface by setting $\hat{x} = x_0$, and is the foundation of many machine learning classification algorithms.

$\endgroup$
  • $\begingroup$ Keep in mind that the question is about not just any continuous predictor, but one that unerringly equals the true probability of $Y = 1$. Using such a predictor for ordinary classification instead of probabilistic classification means throwing away a lot of information. $\endgroup$ – Kodiologist Feb 1 '18 at 17:55
1
$\begingroup$

Not a direct answer to your question, but regarding how Specificity and Sensitivity are used in assessing probabilistic classifiers:

  • Sensitivity is the True Positive Rate (TPR), i.e. Recall.

  • Specificity is the True Negative Rate (TNR), that is 1- False Positive Rate (FPR).

The two can be used in tandem for assessing a probabilistic classifier. Specifically, looking at an ROC curve (TPR plotted versus FPR for different probability thresholds) basically allows you to represent the Sensitivity and 1-Specificity trade-off for a probabilistic classifier. It is true that thresholding is still applied (so there is a conversion to your 1st binary scenario), but across different cut-off points. So the area under the ROC value is an indicator of systemic performance across different probability cut-offs, and captures the TPR/FPR tradeoff.

enter image description here

$\endgroup$
0
$\begingroup$

In your second scenario (which isn't actually a generalization of the first scenario), it's not clear to me why SEN and SPC "are important measures of the usefulness of the predictor", nor why you would want to assess the usefulness of this predictor at all. Sensitivity and specificity, in the usual senses of the words, are interesting because they quantify how close the relationship is between $X$ and $Y$. But when $X$ tells us exactly $P(Y = 1)$, then surely no further quantification of the relationship is necessary. You have a perfect predictor of $P(Y = 1)$. What could be better? Notice that in practice, we can rarely obtain such a predictor. We get only partly accurate predictors of $P(Y = 1)$, and we can compare the predicted probabilities to actual observed frequencies with proper scoring rules.

$\endgroup$
  • $\begingroup$ Let me clarify how I see a generalization. In the first scenario, $X$ assumes values in the set, $\{0,1\}$. In the second scenario, $X$ can assume values in the unit interval, $[0,1]$. The sample space is the only generalization. Also, if we have a continuous-valued $X$, we can transform it into a binary r.v., $Z$, by letting $Z \mid X=x \sim \textrm{Bernoulli}(x)$. Then, the expectations in the formulas for SEN and SPC are unaffected by replacing $X$ with $Z$ and we can rightfully refer to the sensitivity and specificity of $Z$ because it's binary. $\endgroup$ – jjet Jan 26 '18 at 20:45
  • $\begingroup$ @jjet I see, but it is not the case that for any $X$ under scenario 1, there could exist a continuous variable that round-trips back to $X$ through that transformation. In other words, scenario 1 is not a special case of scenario 2, and hence scenario 2 is not a generalization of scenario 1. In fact, scenario 1 is the broader case because $X$ and $Y$ could have any of a variety of degrees of association, whereas scenario 2 requires perfect association. $\endgroup$ – Kodiologist Jan 26 '18 at 20:49
  • $\begingroup$ I only intended to refer to the sample space in noting the generalization. If I understand your comment correctly, it's that there exists no mapping from a binary set onto a continuous set. I get that point although such a mapping can occur by introducing a randomization procedure. But I only brought that up to underscore the fact that a test (however crappy) can be generated from $X$ which has a well-defined sensitivity and specificity. Moreover, the same expectations for $Z$ are equivalent to those for $X$. It stands to reason that $X$ could then have a kind of sensitivity and specificity. $\endgroup$ – jjet Jan 26 '18 at 20:57
  • $\begingroup$ To your other point, I have a predictor, $X$, of a binary r.v., $Y$ which is much better in some situations than others. If $Y=0$, then $X$ is almost always very close to $0$. Unfortunately, if $Y=1$, then, $E(X|Y=1)=.3$ and $X$ has quite a bit of variation in this case. In a general sense, I simply wanted to note that $X$ was very "specific" but not very "sensitive." $\endgroup$ – jjet Jan 26 '18 at 21:00
  • $\begingroup$ @jjet Regarding your bounty, I'm not sure what statement in my answer requires a citation or what kind of sources you're hoping to find. $\endgroup$ – Kodiologist Jan 29 '18 at 16:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.