Is it valid to analyze signal detection data without employing metrics derived from signal detection theory?

Question

A signal detection experiment typically presents the observer (or diagnostic system) with either a signal or a non-signal, and the observer is asked to report whether they think the presented item is a signal or non-signal. Such experiments yield data that fill a 2x2 matrix:

Signal detection theory represents such data as representing a scenario where the "signal/non-signal" decision is based on a continuum of signal-ness on which the signal trials generally have a higher value than the non-signal trials, and the observer simply chooses a criterion value above which they will report "signal":

In the diagram above, the green and red distributions represents the "signal" and "non-signal" distributions, respectively and the grey line represents a given observer's chosen criterion. To the right of the grey line, the area under the green curve represents the hits and the area under the red curve represents the false alarms; to the left of the grey line, the area under the green curve represens misses and the area under the red curve represents correct rejections.

As may be imagined, according to this model, the proportion of responses that fall into each cell of the 2x2 table above is determined by:

1. The relative proportion of trials sampled from the green and red distributions (base rate)
2. The criterion chosen by the observer
3. The separation between the distributions
4. The variance of each distribution
5. Any departure from equality of variance between distributions (equality of variance is depicted above)
6. The shape of each distribution (both are Gaussian above)

Often the influence of #5 and #6 can only be assessed by getting the observer to make decisions across a number of different criteria levels, so we'll ignore that for now. Additionally, #3 and #4 only make sense relative to one another (eg. how big is the separation relative to the variability of the distributions?), summarized by a measure of "discriminability" (also known as d'). Thus, signal detection theory proscribes estimation of two properties from signal detection data: criterion & discriminability.

However, I have often noticed that research reports (particularly from the medical field) fail to apply the signal detection framework and instead attempt to analyze quantities such as "Positive predictive value", "Negative predictive value", "Sensitivity", and "Specificity", all of which represent different marginal values from the 2x2 table above (see here for elaboration).

What utility do these marginal properties provide? My inclination is to disregard them completely because they confound the theoretically independent influences of criterion and discriminability, but possibly I simply lack the imagination to consider their benefits.

Answer 1

The Positive Predictive Influence (PPV) is not a good measure, not only because it confounds both mechanisms (discriminability and response bias), but also because of item base-rates. It is preferable to use the posterior probabilities, like P(signal|"yes"), which account for item base-rates:

$P(signal|yes) = \frac{P(signal)P(Hit)}{P(signal)P(Hit)+P(noise)P(False Alarm)}$

but... what is it good for?? well, it is useful for adjusting the response criteria in order to maximize/minimize a the probability of a specific outcome. So, it is complementary to the sensitivity and response bias measures in the sense that it helps to summarize the outcomes of changes in response bias.

A word of advice: if you are sticking with 2x2 outcome matrix that basically only allows you to get a sensitivity measure like d', don't even bother with SDT and just use Hits-False Alarms. Both measures (d' and (H-F)) have a correlation of .96 (no matter what BS detection theorists might come up with)

hope this helps cheers

Answer 2

You're comparing "What is the probability that a positive test outcome is correct given a known prevalence and test criterion?" with "What is the sensitivity and bias of an unknown system to various signals of this type?"

It seems to me that the two both use some similar theory but they really have very different purposes. With the medical tests criterion is irrelevant. It can be set to a known value in many cases. So, determining the criterion of the test is pointless afterwards. Signal detection theory is best for systems where criterion is unknown. Furthermore, prevalence, or signal, tends to be a fixed (and often very small) value. With SDT you often work out a mean d' over varying signals modelling a very complex situation as a few simple descriptors. When both the criterion and signal are fixed known quantities can SDT tell you anything interesting? It seems like a lot of mathematical sophistication to deal with a fundamentally simpler problem.

Answer 3

This might be an over-simplification, but specificity and sensitivity are measures of performance, and are used when there isn't any objective knowledge of the nature of the signal. I mean your density vs. signalness plot assumes one variable that quantifies signalness. For very high dimensional, or infinite-dimensional data, and without a rigorous, provable theory of the mechanism of the generation of the signal, the selection of the variable is non-trivial. The question then arises, is why, after selecting such a variable, are its statistical properties, like the mean and variance for the signal and non-signal not quantified. In many cases, the variable may not be just normal, Poisson, or exponentially distributed. It may even be non-parametric, in which case quantifying the separation as mean difference over variance etc., does not make much sense. Also, a lot of literature in the biomedical field is focussed on applications, and ROC, specificity-sensitivity etc., can be used as objective criteria for comparing the approaches in terms of the limited nature of the problem, and basically that is all that is required. Sometimes people may not be interested in describing, say the actual discrete version log-gamma distribution of the ratio of gene1 vs gene2 transcript abundance in diseased vs control subjects, rather the only thing of importance is whether this is elevated and how much variance of the phenotype or probability of disease it explains.