2
$\begingroup$

What is lost/missed out on if defining d', the sensitivity index from Signal Detection Theory, based on non-standardised rates?

For example, Patel et al. 2008, for a task where normal and anomalous chords have to be discriminated, define performance as simply the HR-FAR difference, instead of z(HR)-z(FAR), although granted they don't claim that this difference represents d'.

enter image description here

It seems that if using non-standardised (hit and false-alarm) rates as in this paper, nothing changes with regards to chance level responding, which is still 0, just as in the case of standardised rates (d'). However, ceiling-level responding, i.e. maximal performance (hits 100%, false alarms 0%), now neatly corresponds to 1, whereas with standardised rates, the ceiling d' depends on which approximation one chooses to avoid the division by zero problem (details here), but is in either case a non-round (and 'arbitrary-looking') number.

Assuming the advantage of using non-standardised rates is in fact this - namely having a convenient range of performance values (0 and 1) for chance and ceiling respectively - then the question is: is there also a disadvantage or cost to non-standardised rates?


Reference:

Patel, A. D., Iversen, J. R., Wassenaar, M., & Hagoort, P. (2008). Musical syntactic processing in agrammatic Broca’s aphasia. Aphasiology, 22(7–8), 776–789. https://doi.org/10.1080/02687030701803804

$\endgroup$

1 Answer 1

0
+50
$\begingroup$

(I would have thought this was covered in my answer to your prior question, and the additional threads linked therein.)

The analysis in the linked paper seems poor to me. I've only skimmed it, but I can't even find where the actual hit and false alarm rates are listed. As noted previously, looking only at the difference between H and Fa conflates people's ability to detect the difference and their tendency to respond. It is easy to see that the same difference between H and Fa can mean that people find the task easier or harder, and that they are more or less likely to respond 'yes'. First, recall the formulas for converting H and Fa into the receiver's sensitivity to the signal and the position of the criterion:

\begin{align} d' &= \quad\ \ \ z(H) - z(Fa) \\[10pt] C &= -\Bigg(\frac{z(H) + z(Fa)}{2}\Bigg) \end{align}

Using these formulas, we can compare difference to the detectability and bias with different rates (coded in R):

d         = data.frame(H  = c(.15, .35, .55, .75, .60),
                       Fa = c(.05, .25, .45, .65, .40) )
d$diff    = d$H-d$Fa
d$d.prime = round( qnorm(d$H)-qnorm(d$Fa), 2)
d$C       = round( -( (qnorm(d$H)+qnorm(d$Fa))/2 ), 2)
d
#      H   Fa diff d.prime     C
# 1 0.15 0.05  0.1    0.61  1.34
# 2 0.35 0.25  0.1    0.29  0.53
# 3 0.55 0.45  0.1    0.25  0.00
# 4 0.75 0.65  0.1    0.29 -0.53
# 5 0.60 0.40  0.2    0.51  0.00

As we move from row 1 to 3, the bias (strictly speaking the placement of the criterion) decreases, but d' also decreases, even though the raw difference remains constant. Comparing rows 2 and 4, we see that the difference is the same and d' is the same, but the bias differs from being reluctant to say 'yes' to being overeager to do so. If we compare rows 1 and 5, we see that the difference between H and Fa can be larger and yet the receiver is not as good at differentiating between the stimulus and noise.

If a receiver gave the maximal performance (hits 100%, false alarms 0%), your task is too easy to determine the receiver's sensitivity. That is a problem with your study, not the metric. I don't follow your reference to a problem with dividing by zero, and I don't see it on the linked page (but again, I just skimmed it). At any rate, what you are calling "non-standardized rates" should not be used because they conflate different aspects of responding.

$\endgroup$
2
  • $\begingroup$ Thank you, and apologies that (as I now realise) your answer to my other question (which was in fact similar) did indeed address the point. The extra example you provide now is however very helpful in illustrating how a constant HR-FAR difference can correspond not just to different response biases but also do different d' values. The only small thing I still don't understand is the role played in all this by z-scoring the two rates; that is, intuitively, how exactly does ignoring the N(0,1) distribution in transforming the scores lead to conflating the different aspects of responding. $\endgroup$
    – z8080
    Dec 18, 2019 at 8:05
  • 1
    $\begingroup$ @z8080, it's probably not best to call it "z-scoring". As I discussed in my other answer, turning data into z-scores is a linear transformation. Here, taking $z({\rm rate})$ (or better: $\Phi^{-1}({\rm rate})$) is a non-linear transformation. It may help you to focus on the figures in the SDT introductions you've seen, or, in a different context, you can see my answer to PP-plots vs. QQ-plots, which passes values through a normal CDF. $\endgroup$ Dec 18, 2019 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.