Generalized Sensitivity and Specificity I'm interested in notation and terminology regarding performance measures related to Sensitivity (SEN) and Specificity (SPC). These two measures are commonly used in binary classification problems. I'm interested in analogous measures for probabilistic classification.
$\textbf{Standard Scenario (Binary Classification):}$
Let $Y$ be a binary response variable where $Y=0$ is a "negative" result and $Y=1$ is a "positive" result. Let $X\in\{0,1\}$ be a test aimed at predicting $Y$. Then, it can be shown that
$$\mathrm{SEN} = \dfrac {E[XY]} {E[Y]}$$
$$\mathrm{SPC} = \dfrac {E[(1-X)(1-Y)]} {E[1-Y]}$$
*I realize these measures are typically defined with different notation using empirical counts rather than as expectations of random variables.
$\textbf{Generalized Scenario (Probabilistic Classification):}$
Let $Y$ be the same as before but now $X\in[0,1]$ is a continuous random variable such that $P(Y=1 \mid X=x)=x$. Certainly, $\mathrm{SEN}$ and $\mathrm{SPC}$ - as defined above - are useful measures of the quality of the predictor, $X$. 
$\textbf{Questions:}$


*

*Is it acceptable to refer to these quantities as the Sensitivity and Specificity of $X$ in the generalized case? Or are different names and considerations warranted when $X$ is a probabilistic classifier?

*Are there any references to this (or a related) concept in the academic literature? I'm struggling to find any.

 A: Thinking in terms of decision theory can help you make continuous analogues out of the quantities of sensitivity and specificity. 
Bishop pp. 40 illustrates a binary classification problem along a single (continuous) input dimension, using a stylized representation of joint probabilities $p(c,C_k)$ for each of 2 classes:

If you are classifying a continuous $x$ into one of two classes $C_1, C_2$ then you are effectively drawing a decision boundary/surface $\hat{x}$ in such a way that the input space for $x$ is divided into decision regions $R_1, R_2$. The true decision boundary $x_0$ is unknown.
The analogous terms to "true positive" and "true negative" would be the probability of correctly classifying a point in either $C_1$ or $C_2$, depending on what those $C_k$ actually represent. In the chart above, they are represented by white regions under the curve. To evaluate them, 


*

*The probability of correctly classifying a point in $C_1$ is $\int_{R_1} p(x,C_1)dx$

*The probability of correctly classifying a point in $C_2$ is $\int_{R_2} p(x,C_2)dx$


You could also calculate the related "false positive" and "false negative" quantities, which are represented by the coloured red, green & blue areas under the curve:


*

*The probability of incorrectly classifying a point in $C_1$ as $C_2$ is $\int_{R_2} p(x,C_1)dx$

*The probability of incorrectly classifying a point in $C_2$ as $C_1$ is $\int_{R_1} p(x,C_2)dx$


Then sensitivity and specificity would simply be the appropriate ratios of those regions, based on which of $C_1,C_2$ represents the "positive" and which represents the "negative":
$\text{sensitivity} = \frac{\text{true positives}}{\text{true positives} + \text{false negatives}}$
$\text{specificity} = \frac{\text{true negatives}}{\text{true negatives} + \text{false positives}}$
This isn't just a theoretical observation. In our example, the green & blue areas under the curve cannot be reduced, but it should be possible to minimise the red area by minimising the expected loss. This actually solves for the optimal decision boundary/surface by setting $\hat{x} = x_0$, and is the foundation of many machine learning classification algorithms.
A: Not a direct answer to your question, but regarding how Specificity and Sensitivity are used in assessing probabilistic classifiers:


*

*Sensitivity is the True Positive Rate (TPR), i.e. Recall.

*Specificity is the True Negative Rate (TNR), that is 1- False Positive Rate (FPR).
The two can be used in tandem for assessing a probabilistic classifier. Specifically, looking at an ROC curve (TPR plotted versus FPR for different probability thresholds) basically allows you to represent the Sensitivity and 1-Specificity trade-off for a probabilistic classifier. It is true that thresholding is still applied (so there is a conversion to your 1st binary scenario), but across different cut-off points. So the area under the ROC value is an indicator of systemic performance across different probability cut-offs, and captures the TPR/FPR tradeoff.

A: In your second scenario (which isn't actually a generalization of the first scenario), it's not clear to me why SEN and SPC "are important measures of the usefulness of the predictor", nor why you would want to assess the usefulness of this predictor at all. Sensitivity and specificity, in the usual senses of the words, are interesting because they quantify how close the relationship is between $X$ and $Y$. But when $X$ tells us exactly $P(Y = 1)$, then surely no further quantification of the relationship is necessary. You have a perfect predictor of $P(Y = 1)$. What could be better? Notice that in practice, we can rarely obtain such a predictor. We get only partly accurate predictors of $P(Y = 1)$, and we can compare the predicted probabilities to actual observed frequencies with proper scoring rules.
