5
$\begingroup$

I have a series of scores in a signal detection task. For each block of scores (i.e. a set of scores from one participant on one day) I have calculated a d' score which I am using as an indicator of performance.

These d' scores rise over time, which is an interesting result. Is there any way I can calculate whether the change is statistically significant?

$\endgroup$
3
  • $\begingroup$ if these are human participants, the increased performance would be quite typical, and attributed to a practice effect. $\endgroup$ Jun 1, 2011 at 21:17
  • $\begingroup$ That's what I'm looking for yes. The task in question is very hard though, and I'm looking for a way to quantify/categorise the improvement. $\endgroup$
    – Tom Wright
    Jun 1, 2011 at 22:08
  • $\begingroup$ Do you have multiple participants or just a single participant? How many blocks do you have? $\endgroup$ Jun 2, 2011 at 6:15

2 Answers 2

4
$\begingroup$

I've come to decide that the best approach to the analysis of signal detection data (frankly, any data with dichotomous stimuli & responses) collected in multiple participants is to use a generalized mixed model, treating participant as a random effect and predicting response as a function of truth and whatever other explanatory variables are in the experiment. Effects that involve the variable specifying the truth reflect effects on in discriminability, while effects not involving that variable reflect effects on response bias.

For example, say you present a list of items for a participant to remember, then later present them with a second list containing some items they were asked to remember as well as some new items, asking the participant to label each as "old" or "new". You use words of different concreteness, and want to determine whether word concreteness affects discrimination ability. Thus, you would have data as:

participantID    word    concreteness    truth    response
1                brick   10              old      old
1                happy   2               old      new
1                river   8               new      new
1                peace   1               old      old
...

You could fit the model (first converting "response" to 0/1) using the lmer function from the lme4 package in R:

my_data$response_num = as.numeric(factor(my_data$response))-1
my_mix = lmer(
    formula = response_num ~ (1|participant) + truth*concreteness
    , family = binomial
    , data = my_data
)
print(my_mix)

Or, if you want likelihood ratios (and you should!), you can use ezMixed function from the ez package in R:

my_mix = ezMixed(
    data = my_data
    , dv = .(response_num)
    , random = .(participant)
    , fixed = .(truth, concreteness)
    , family = binomial
)
print(my_mix$summary)

In both approaches (ezMixed is simply a wrapper around lmer, but with additional computation of likelihood ratios), the intercept reflects any overall bias in labelling words as new/old. The main effect of truth reflects the discriminability of new/old words. The main effect of concreteness reflects any effect of concreteness on response bias. Finally, the truth:concreteness interaction reflects any effect of concreteness on discriminability of new/old words.

A couple points about this case specifically. Since this example deals with lexical stimuli, it may be reasonable to model words as a random effect as well (add + (1|word) to the lmer formula, or add word to the list of random effects in the call to ezMixed). Additionally, the model above fits a linear function to the effects involving concreteness. If you want to account for non-linearity, you might employ generalized additive mixed models (implemented in the gamm4 package). Unfortunately ezMixed currently only handles non-linearity by permitting polynomials up to a user-specified degree, which I feel is less useful than GAMM. Adding GAMM to ezMixed is on my to do list...

$\endgroup$
4
  • $\begingroup$ I forgot to mention that the measures of bias & discriminability in this approach are on the log-odds scale. Thus, the estimate of the intercept is the log-odds of responding 'old', the estimate of the main effect of truth is the increased rate of responding 'old' (in log-odds) induced by old-ness, etc. $\endgroup$ Jun 1, 2011 at 23:23
  • $\begingroup$ Oops, I just realized that my description implies direct interpretation of the estimates from an lmer model, whereas I was more thinking about the ezMixed results. If interpreting the lmer estimates, you need to take into account what type of contrasts were used when creating the model. By default R uses treatment contrasts, but my interpretations above are appropriate to sum contrasts (eg. intercept means overall bias, truth effect reflects discriminability, etc). $\endgroup$ Jun 1, 2011 at 23:28
  • 1
    $\begingroup$ Yet another comment on my own answer: I somehow managed to miss the fact that your question states a particular interest in effects of time. You can of course add time as a variable to the above demonstrated approach in the same manner as I use concreteness as a variable. In fact, if you have blocks and trials-within-blocks, you might reasonably add both block and trial as predictors. If your task is somewhat demanding but participants get rest between blocks, I'd argue that effects of block characterize effects of practice while effects of trial characterize effects of fatigue. $\endgroup$ Jun 2, 2011 at 12:36
  • $\begingroup$ While this approach seems interesting, I'm not sure if it directly answers how to directly interpret changes in d' or response bias. For example: what meaning does an $L_p$ distance of d' have ? Or is it pointless to say literally compute the difference of d' s or mean square error? $\endgroup$
    – Mecasickle
    Jun 7, 2018 at 7:21
1
$\begingroup$

The appropriate analysis would depend on whether you have multiple participants or just one participant and the number of blocks that you have. In general, with more blocks, you can more precisely characterise the functional relationship between practice and performance.

Small number of blocks (e.g., 3 to perhaps 10)

  • Linear and quadratic contrasts as part of a repeated measures ANOVA would provide a basic test of the improvement in d-prime with practice. Or you could implement the same model within a mixed-model framework. Another simple option would be to compare the first one or two blocks with the last one or two blocks either using a repeated measures t-test or using a contrast with appropriate weights as part of the repeated measure ANOVA.

Many blocks (e.g., perhaps 15 or 30 or more)

  • You could start to characterise the change in d-prime with practice more precisely perhaps with some non-linear functions. Practice effects are usually more rapid at the start of practice and monotonically decelerate and approach an asymptote. Thus, non-linear regression per participant or non-linear multilevel modelling for a more integrated approach represent two major options.

Distribution of residuals for d-prime

  • If you had concerns about the distribution of residuals of d-prime then you could consider a transformation; I'm not sure what's standard practice in signal detection research, but at a guess I thought d-primes might be relatively normal without transformation.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.