Is there a way to determine the significance of a change in a d' score?

Question

I have a series of scores in a signal detection task. For each block of scores (i.e. a set of scores from one participant on one day) I have calculated a d' score which I am using as an indicator of performance.

These d' scores rise over time, which is an interesting result. Is there any way I can calculate whether the change is statistically significant?

Answer 1

I've come to decide that the best approach to the analysis of signal detection data (frankly, any data with dichotomous stimuli & responses) collected in multiple participants is to use a generalized mixed model, treating participant as a random effect and predicting response as a function of truth and whatever other explanatory variables are in the experiment. Effects that involve the variable specifying the truth reflect effects on in discriminability, while effects not involving that variable reflect effects on response bias.

For example, say you present a list of items for a participant to remember, then later present them with a second list containing some items they were asked to remember as well as some new items, asking the participant to label each as "old" or "new". You use words of different concreteness, and want to determine whether word concreteness affects discrimination ability. Thus, you would have data as:

participantID    word    concreteness    truth    response
1                brick   10              old      old
1                happy   2               old      new
1                river   8               new      new
1                peace   1               old      old
...

You could fit the model (first converting "response" to 0/1) using the lmer function from the lme4 package in R:

my_data$response_num = as.numeric(factor(my_data$response))-1
my_mix = lmer(
    formula = response_num ~ (1|participant) + truth*concreteness
    , family = binomial
    , data = my_data
)
print(my_mix)

Or, if you want likelihood ratios (and you should!), you can use ezMixed function from the ez package in R:

my_mix = ezMixed(
    data = my_data
    , dv = .(response_num)
    , random = .(participant)
    , fixed = .(truth, concreteness)
    , family = binomial
)
print(my_mix$summary)

In both approaches (ezMixed is simply a wrapper around lmer, but with additional computation of likelihood ratios), the intercept reflects any overall bias in labelling words as new/old. The main effect of truth reflects the discriminability of new/old words. The main effect of concreteness reflects any effect of concreteness on response bias. Finally, the truth:concreteness interaction reflects any effect of concreteness on discriminability of new/old words.

A couple points about this case specifically. Since this example deals with lexical stimuli, it may be reasonable to model words as a random effect as well (add + (1|word) to the lmer formula, or add word to the list of random effects in the call to ezMixed). Additionally, the model above fits a linear function to the effects involving concreteness. If you want to account for non-linearity, you might employ generalized additive mixed models (implemented in the gamm4 package). Unfortunately ezMixed currently only handles non-linearity by permitting polynomials up to a user-specified degree, which I feel is less useful than GAMM. Adding GAMM to ezMixed is on my to do list...

Answer 2

The appropriate analysis would depend on whether you have multiple participants or just one participant and the number of blocks that you have. In general, with more blocks, you can more precisely characterise the functional relationship between practice and performance.

Small number of blocks (e.g., 3 to perhaps 10)

• Linear and quadratic contrasts as part of a repeated measures ANOVA would provide a basic test of the improvement in d-prime with practice. Or you could implement the same model within a mixed-model framework. Another simple option would be to compare the first one or two blocks with the last one or two blocks either using a repeated measures t-test or using a contrast with appropriate weights as part of the repeated measure ANOVA.

Many blocks (e.g., perhaps 15 or 30 or more)

• You could start to characterise the change in d-prime with practice more precisely perhaps with some non-linear functions. Practice effects are usually more rapid at the start of practice and monotonically decelerate and approach an asymptote. Thus, non-linear regression per participant or non-linear multilevel modelling for a more integrated approach represent two major options.

Distribution of residuals for d-prime

• If you had concerns about the distribution of residuals of d-prime then you could consider a transformation; I'm not sure what's standard practice in signal detection research, but at a guess I thought d-primes might be relatively normal without transformation.