5
$\begingroup$

I have data from an experiment in which participants had to respond to stimuli in two different conditions in a within-subject design. Obviously, I can fairly easily test for differences in accuracy between the two conditions. However, my initial idea was to check for differences in reaction times. When a subject incorrectly answers, I feel the associated reaction time is meaningless. What is the 'standard' way to do this?

$\endgroup$
6
  • 1
    $\begingroup$ If I understand your question, it is something like this: I know how to conduct a paired t test on A. But I want to perform this test on B. But B is sometimes meaningless. What is the standard way to do this? I won't speak for the rest of your audience, but I have no idea what statistical question you are asking. :) Standard way to do what? Can you expand a little bit? $\endgroup$ – Alexis Jun 29 '14 at 15:58
  • $\begingroup$ I want to compare reaction times for choice. If the wrong choice is made, reaction time seems meaningless. How do other people normally, usually, often, most commonly solve this issue. A reaction measure of choice is a very common (standard) experiment, so there must be prior experience out there. $\endgroup$ – LBogaardt Jun 29 '14 at 16:40
  • 1
    $\begingroup$ So please edit your question (not comment) to amplify. $\endgroup$ – Alexis Jun 29 '14 at 16:41
  • 1
    $\begingroup$ Why is the reaction time meaningless if the answer is incorrect? Why are you looking at reaction time? $\endgroup$ – Joel W. Jun 29 '14 at 17:37
  • $\begingroup$ Ok, so imagine trying to select the 'correct' image out of a pair, but it's not easy to tell the difference. Under some condition, it may be easier to spot the correct image. This is my hypothesis. I'd like to check this using subjects' accuracy, and also by looking at the time it takes before they make up their mind (believing the condition makes it easier, and faster, to find the correct image). $\endgroup$ – LBogaardt Jun 30 '14 at 13:38
1
$\begingroup$

The traditional approach here has been to analyse accuracy and speed separately, so you would first compare the two conditions in terms of accuracy (paired-samples t test), and then discard the wrong answers, and compare the reaction times for correct answers from both conditions.

However, as you've said, there are obvious limitations to this approach: speed and accuracy interact in a myriad of ways, including but not limited to the well-known "speed-accuracy trade off": some subjects put more emphasis on accuracy, and respond slowly, some do the opposite.

A huge amount of work has been done on this problem, including the Inverse Efficiency Score referred to by @LGBogaardt. There are alternatives along the same lines - statistical models which take both accuracy and speed into account.

Another option, which I prefer, is to fit a sequential sampling model, such as a diffusion model (there are many other options here, but that paper is a good start). This basically models the process underlying your participants' responses as a simple, semi-random movement over time, which drifts generally towards the right answer, but, because of random noise, sometimes gives the wrong answer instead, and moves less consistently under harder conditions. The great strength of this method is that it allows you to make more informed inferences about the underlying processes, rather than just identifying one condition as "faster".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.