10
$\begingroup$

I would like to calculate d prime for a memory task that involves detecting old and new items. The problem I have is that some of the subjects have hit rate of 1 and/or false alarm rate of 0, which makes the probabilities 100% and 0%, respectively.

The formula for d prime is d' = z(H) - z(F), where z(H) and z(F) are the z transforms of hit rate and false alarm, respectively.

To calculate the z transform, I use the Excel function NORMSINV (i.e., z(H)=NORMSINV(hit rate)). However, if the hit rate or false alarm rate is 1 or 0, respectively, the function returns an error. This is because z transform , as I understand, indicates the area under the ROC curve, which does not mathematically allow for 100% or 0% probability. In this case, I'm not sure how to calculate d' for the subjects with ceiling performance.

One website suggests replacing 1 and 0 rate with 1 - 1/(2N) and 1/2N with N being the maximum number of hits and false alarms. Another website says "neither H nor F can be 0 or 1 (if so, adjust slightly up or down)". This seems arbitrary. Does anyone have an opinion on this or would like to point me to the right resources?

$\endgroup$
6
$\begingroup$

Both sites are suggesting the same thing but one is posing a way to consistently select the amount of adjustment. This has been attributed to a number of people but I don't think anyone knows who really came up with it first. Different fields have a different seminal book or author on signal detection. What's important is that the method you select is reasonable.

The one method you posted usually is taken to imply that if you had a much larger set of items (2N) then you would have been able to detect at least one error. If this is a reasonable way to think about the problem then you're done. I doubt it is for a memory test. In the future you might want to raise N to insure this is much less likely to happen. Nevertheless, the method is salvageable if you consider it a different way. You're adjusting on a hypothetical average of two runs of the same number of memory items. In that case you're saying that in another run of the experiment (assuming new items or they forgot all of the old ones) there would have been an error. Or, more simply, you're just selecting half way between the highest imperfect score that you can measure and a perfect score.

This is a problem with no simple universal solution. The first question you need to ask is whether you believe, in your case, you have genuine perfect classification. In that case your data is your data. If not, then you believe it's just variability in the sample causing hits to be 100%. Once you conclude that's the case then you've got to consider reasonable ways to generate an estimate of what you believe d' to be. And so you have to ask yourself what it actually is.

The easiest way to determine what d' should be is to look at the other data in that same condition. You could perhaps estimate that the accuracy for this one participant is half way between the next best value that you have and 100% (which may turn out to be exactly the same as the value you found). Or, it could be some very small amount greater. Or it could just be equal to the best values. You've got to select what you believe is the best answer based on your data. A more specific question posted might help you here.

You should attempt to insure you do is make as little impact on the criterion as possible. In your case an adjustment to hits and FAs will cause the criterion not to shift at all. However, if you adjust hits when say, FAs = 0.2, then you have to be careful about how that adjustment would impact the interpretation of criterion. You're sort of obligated in that case to make sure hits is very high.

$\endgroup$
  • $\begingroup$ Thank you for your perspective and rationale, John. That really helps. I should add that increasing number of N is a good solution but our task is done inside the fMRI scanner and we're limited by how many trials we can make subjects do before they get fidgety and ruin our data with motion. $\endgroup$ – A.Rainer Jan 24 '15 at 20:50
  • $\begingroup$ "You could perhaps estimate that the accuracy for this one participant is half way between the next best value that you have and 100% (which may turn out to be exactly the same as the value you found)" is good advice. This will preserve the order of the subject performance ranking. $\endgroup$ – A.Rainer Jan 24 '15 at 21:07
13
$\begingroup$

Stanislaw & Todorov (1999) have a good discussion of this under the heading Hit and False-Alarm Rates of Zero or One.

They discuss the pros and cons of several methods for dealing with these extreme values, including:

  • Use a non-parametric statistic such as $A'$ instead of $d'$ (Craig, 1979)

  • Aggregate data from multiple subjects before calculating the statistic (Macmillan & Kaplan, 1985)

  • add 0.5 to both the number of hits and the number of false alarms, and add 1 to both the number of signal trials and the number of noise trials; dubbed the loglinear approach (Hautus, 1995)

  • Adjust only the extreme values by replacing rates of 0 with $0.5/n$ and rates of 1 with $(n-0.5)/n$ where $n$ is the number of signal or noise trials (Macmillan & Kaplan, 1985)

The choice is ultimately up to you. Personally I prefer the third approach. The first approach has the drawback that $A'$ is less interpretable to your readers who are much more familiar with $d'$. The second approach may not be suitable if you are interested in single-subject behavior. The fourth approach is biased because you are not treating your data points equally.

$\endgroup$
  • $\begingroup$ MacMillan & Kaplan is a popular reference but I believe there's a Greek or Russian author that predates them for the same advice... name is too long for me to remember. My quick look at S&T is that it's not very comprehensive but it does remind me that I was probably mistaken not mentioning A'. Of course, once you do that you have to explain what it really means switching to non-parametric. $\endgroup$ – John Jan 24 '15 at 22:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.